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May  10, 1993 


The  Honorable  Ronald  V.  Dellums 
Chairman,  Committee  on  Armed  Services 
House  of  Representatives 

Dear  Mr.  Chairman: 

In  response  to  your  request,  this  report  describes  the  evolving  coordination  of  simulation 
activities  under  the  auspices  of  the  Defense  Modeling  and  Simulation  Office  within  the  Office  of 
the  Secretary  of  Defense.  It  also  addresses  the  Army’s  efforts  to  strengthen  the  management  of 
its  simulation  programs,  with  a  particular  focus  on  the  acquisition  of  the  Close  Combat  Tactical 
Trainer.  This  report  contains  recommendations  to  the  Secretaries  of  Defense  and  the  Army. 

We  are  sending  copies  of  this  report  to  the  House  and  Senate  Committees  on  Appropriations; 
the  Chairman  of  the  Senate  Committee  on  Armed  Services;  the  Director,  Office  of  Management 
and  Budget;  and  the  Secretaries  of  the  Army,  Navy,  and  Air  Force.  Copies  will  also  be  made 
available  to  other  interested  parties  upon  request. 


This  report  was  prepared  under  the  direction  of  Paul  L.  Jones,  who  can  be  reached  on 
(202)  512-3990.  Other  msqor  contributors  are  listed  in  appendix  III. 


Sincerely  yours, 


Frank  C.  Conahan 
Assistant  Comptroller  General 
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Executive  Summary 


Purpose 


Congress  has  supported  increased  use  of  simulation  technology  but  has 
had  some  concerns  about  the  Department  of  Defense’s  (dod)  management 
of  this  area.  Accordingly,  the  Chairman  of  the  House  Armed  Services 
Committee  asked  gao  to  study  dod’s  efforts  to  coordinate  simulation 
technology’s  use  agencywide.  The  Chairman  also  asked  gao  to  review  the 
Army’s  Close  Combat  Tactical  Trainer  (cctt)  acquisition  plan  to  ensure 
that  it  provided  for  (1)  high-level  management  oversight,  (2)  system 
interoperability,  and  (3)  adequate  integration  of  the  system  with  more 
traditional  forms  of  training  for  greater  cost-effectiveness. 


Background 


Although  dod  uses  simulations  for  a  variety  of  purposes,  this  report 
focuses  on  simulations  in  military  training.  Advances  in  simulation 
technology  are  bringing  about  what  many  consider  to  be  a  revolution  in 
military  training.  In  recent  years,  however,  Congress  and  various  study 
groups  have  expressed  concern  about  the  lack  of  Defense-wide  guidance, 
coordination,  and  management  of  simulation  technology. 

On  November  30, 1992,  the  Army  awarded  a  contract  to  initiate  the 
engineering  design  of  the  cctt — a  simulation  system  to  train  armor  and 
mechanized  infantry  forces,  cctt  is  the  first  of  a  planned  series  of 
simulators  to  train  armor,  aviation,  air  defense,  artillery,  and  engineer 
forces.  The  series  is  called  the  Combined  Arms  Tactical  Training  system. 
cctt  is  expected  to  have  an  acquisition  cost  of  over  $400  million  and  a 
15-year  life-cycle  cost  of  nearly  $1.3  billion.  Current  milestone  dates  call 
for  decisions  on  full  production  to  be  made  in  1997. 


Results  in  Brief 


dod’s  coordination  of  the  development  and  use  of  simulation  technology  is 
still  evolving.  The  Defense  Modeling  and  Simulation  Office  (dmso)  and  the 
Joint  Staff  are  key  players,  dmso  is  working  to  foster  increased  use  of 
simulation  technology  across  a  broad  spectrum  of  functions  affecting  the 
military  services;  it  is  focusing  particularly  on  the  linking  of  diverse 
simulations.  Two  Joint  Staff  directorates  support  the  simulation 
requirements  of  regional  commanders  in  chief  (cincs)  who  report  to  the 
Secretary  of  Defense  through  the  Chairman  of  the  Joint  Chiefs  of  Staff. 
dmso  has  made  progress  in  working  to  enhance  dod  simulation 
capabilities,  but  its  future  is  clouded  by  a  lack  of  permanent  staff.  The  two 
Joint  Staff  directorates  have  overlapping  responsibilities  in  the  simulation 
area  and  interpret  them  differently.  This  has  resulted  in  some  duplication 
of  effort  The  Joint  Staff  has  begun  a  reassessment  of  its  organization, 
which  provides  an  opportunity  to  eliminate  unnecessary  overlap. 
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The  Army  recently  reorganized  its  command  structure  to  ensure  high-level 
management  oversight  over  the  acquisition  of  cctt  and  other  simulation 
programs.  It  also  mandated  that  cctt  be  interoperable  with  other 
simulation  systems. 

The  Army  expects  cctt  to  complement  and  in  some  instances  enhance 
training  traditionally  done  in  the  field.  It  expects  that  the  ccrrr’s  cost  will 
be  offset  by  reductions  in  traditional  training  funds.  However,  life-cycle 
cost  data  on  the  system  are  tentative  for  now,  and  the  most  appropriate 
mix  of  ccrr  and  traditional  field  training  has  yet  to  be  determined.  The 
Army  plans  a  more  definitive  assessment  of  these  issues  before  ccrr’s  1997 
full-production  decision.  Even  then,  future  system  upgrades  are  planned 
that  could  affect  cost  and  training  effectiveness. 


Principal  Findings 


DOD-Wide  Coordination  Is  In  June  1991,  the  Deputy  Secretary  of  Defense  authorized  the  creation  of 
Still  Evolving  DMSO  to  coordinate  and  provide  guidance  for  simulation  issues  DOD-wide. 

dmso  has  made  progress  responding  to  its  mandate.  It  has  formulated  a 
management  plan  and  helped  organize  Defense-wide  working  groups  to 
facilitate  coordination  between  the  services  and  dod  organizations.  The 
management  plan  outlines  two  important  areas  that  require  attention:  (1)  a 
master  plan  to  guide  simulation  development  and  use  across  dod  and 
(2)  measures  to  ensure  that  the  services  implement  adequate  controls  over 
the  development  and  modifications  of  simulations.  In  fiscal  year  1993, 
dmso  expects  to  issue  a  dod  directive  outlining  requirements  for  a 
coordinated  approach  for  developing  and  using  simulations.  A  draft  of  the 
directive  encourages  information  sharing,  investments  in  common 
technologies,  and  the  formulation  of  common  standards  for  simulation 
development 

dmso  faces  challenges,  however.  To  date,  the  office  has  been  unsuccessful 
in  obtaining  permanent  staffing  authority  and  hiring  a  civilian  director. 
Staff  temporarily  on  loan  from  the  services  are  starting  to  be  recalled  for 
other  assignments,  which  could  create  gaps  in  institutional  knowledge  and 
hamper  mission  continuity. 

Representatives  from  two  Joint  Staff  directorates  have  worked  with  dmso 
in  various  working  groups.  However,  within  the  Joint  Staff,  the  roles  and 
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responsibilities  concerning  simulation  support  for  cincs  are  not  clearly 
delineated.  For  example,  both  directorates  claim  some  responsibility  for 
assessing  cinc  simulation  requirements,  and  officials  in  each  have  voiced 
different  perspectives  on  how  it  should  be  done.  Further,  each  directorate 
has  been  surveying  the  cincs  to  determine  their  simulation  needs.  This 
results  in  duplication  and  inefficient  use  of  resources.  However,  the  Joint 
Staff  has  begun  to  reassess  various  aspects  of  its  organizational  staffing 
and  structure.  These  reassessments,  expected  to  be  completed  in  fiscal 
year  1993,  provide  opportunities  to  consolidate  functions  and  eliminate 
duplication. 


Army  Acts  to  Ensure 
High-Level  Management 
Oversight  and  System 
Interoperability 


Army  plans  call  for  cctt  and  subsequent  simulations  in  the  Combined 
Arms  Tactical  Training  system  to  each  be  acquired  as  a  major  program  and 
subjected  to  specific  milestone  reviews  by  Army  executives.  To  implement 
this  plan,  the  Army  recently  reorganized  its  command  structure  for 
acquiring  simulations.  It  created  the  Simulation,  Training,  and 
Instrumentation  Command  and  designated  a  project  manager  to  oversee 
development,  testing,  and  fielding  of  ccrr  and  other  related  systems. 

To  ensure  interoperability  among  simulation  systems,  the  Army  required 
that  cctt  have  an  “open  systems  design.”  In  such  a  design,  neither 
interfaces  between  pieces  of  hardware  nor  interfaces  between  sections  of 
software  depend  on  vendor-specific  or  proprietary  equipment  Also,  the 
Army  is  developing  operational  standards  for  cctt  that  could  be  useful  for 
other  simulation  systems,  including  those  of  other  military  services. 


The  Army  Is  Uncertain 
About  the  Most 
Cost-Effective  Mix  of 
Simulation  and  Traditional 
Field  Training 


In  October  1991,  an  executive  review  council  approved  the  engineering 
design  of  cctt.  The  council  based  its  approval  on  preliminary, 
yet-to-be-validated  data  on  likely  system  costs  and  training  effectiveness. 
Cost  and  training  effectiveness  are  scheduled  to  be  reassessed  before  the 
Army  decides  whether  to  proceed  with  the  full-rate  production  of  cctt  in 
1997.  Even  then,  however,  some  uncertainties  could  remain  because  a 
number  of  upgrades  are  planned  for  the  system. 


Army  commanders  are  trying  to  accelerate  the  ccrr  acquisition.  Under  a 
“Quick  Start”  program,  the  Army  decided  to  field  68  limited-capability  cctt 
units  before  it  tested  system  operations  or  validated  cost  and  training 
effectiveness.  Continued  pressures  to  expedite  fielding  are  likely  and 
could  cause  the  Army  to  limit  planned  testing  and  related  assessments. 
However,  shortcutting  these  critical  reviews  could  limit  (1)  the  Army’s 
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ability  to  more  definitively  assess  the  effectiveness  of  cctt,  (2)  full 
disclosure  of  future  program  costs  for  cctt,  and  (3)  insight  into  potential 
funding  trade-offs  between  simulations  and  field  training. 

The  Army  does  not  definitively  know  the  combination  and  quantity  of 
Combined  Arms  Tactical  Training  systems  it  will  need  or  the  extent  to 
which  they  will  be  linked  together  to  train  higher  echelons.  Plans  for  ccrr 
and  other  Combined  Arms  Tactical  Training  systems  initially  focused  on 
integrating  simulations  and  field  training  at  the  battalion  level.  However, 
Army  officials  now  plan  to  use  ccrr  at  the  platoon  and  company  level. 


Recommendations  G*°  recommends  that  the  SeereUiy  of  Defense 

•  ensure  that  dmso  is  properly  staffed  to  carry  out  its  assigned 
responsibilities  and 

•  eliminate  the  overlap  between  the  two  Joint  Staff  directorates  by  clearly 
delineating  each  directorate’s  roles  and  responsibilities  in  the  simulation 
area. 

gao  also  recommends  that  the  Secretary  of  the  Army  ensure  that  all 
testing,  cost  analyses,  and  training  effectiveness  assessments  are 
completed  and  fully  considered  before  decisions  are  made  about  full-rate 
production  of  cctt. 


Agency  Comments 
and  GAO’s  Evaluation 


dod  concurred  with  the  majority  of  gao’s  findings  and  recommendations 
on  dmso  and  the  Army’s  management  of  the  cctt  program  (see  app.  n  for 
complete  dod  comments).  However,  concerning  the  recommendation  to 
eliminate  duplicative  responsibilities  within  two  directorates  of  the  Joint 
Staff,  dod  stated  that  modeling  and  simulation  responsibilities  are  shared 
by  several  joint  staff  directorates,  not  just  those  cited  in  this  report  dod 
expressed  the  view  that  this  does  not  constitute  unneeded  duplication  of 
effort 


Since  gao  found  unnecessary  overlapping  functions  between  two  Joint 
Staff  directorates,  that  resulted  in  inefficient  use  of  resources,  dod’s 
acknowledgement  that  even  more  joint  staff  directorates  have  modeling 
and  simulation  responsibilities  does  not  dispel  gao’s  concern,  but 
reinforces  it  Moreover,  for  dod  to  express  the  opinion  that  there  is  no 
unnecessary  duplication  before  its  own  studies  of  the  issue  are  completed 
seems  to  undermine  the  purpose  of  the  studies. 
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Chapter  1 _ 

Introduction 


The  nature  of  modem  warfare  demands  that  air,  land,  and  sea  forces 
complement  each  other  as  a  team.  U.S.  forces  are  no  longer  preoccupied 
primarily  with  training  for  a  well-defined  threat  in  Europe.  Instead,  they 
must  be  prepared  to  respond  to  multifaceted  crises  that  could  develop 
quickly  in  any  part  of  the  world.  This  requires  repetitive,  realistic,  and 
stressful  training  in  as  close  to  combat  conditions  as  is  permissible  in  a 
peacetime  environment 

Increasingly,  advances  in  computer  technology  are  facilitating  the  ability 
of  simulations  to  complement  and  in  some  cases  offer  significant 
advantages  over  traditional  forms  of  training. 


Training  Simulations 


The  services  traditionally  have  used  hundreds  of  training  devices  to  model 
or  simulate  various  aspects  of  combat  weapon  systems,  and  terrain  in 
support  of  training  activities.  Training  devices  range  from  the 
simple — such  as  simulated  explosives  and  plywood  terrain  boards  that 
replicate  the  terrain  of  a  given  battle  area— to  the  highly  technical— such 
as  sophisticated  laser  gunnery  systems  that  simulate  the  effects  of 
direct-fire  weapons  and  computer-supported  multimillion  dollar  aircraft 
simulators. 


Computer  simulations  are  growing  in  importance  as  training  devices 
because  they  add  realism  to  training  and  potentially  reduce 
training-related  costs.  During  the  1970s,  computer  technology  facilitated 
the  development  of  multimillion  dollar  weapon  system  trainers  that 
generally  focused  on  developing  individual  skills.  For  example,  flight 
simulators  were  used  to  train  pilots.  As  microprocessors  became  widely 
available,  Department  of  Defense  (dod)  components  sought  more 
comprehensive  ways  to  use  this  technology  to  train  their  forces. 
Technology  achievements  in  the  1980s  permitted  the  networking  of 
multiple,  homogeneous  weapon  system  simulators  into  an  interactive, 
electronic  battlefield  where  military  crews  could  obtain  realistic  training 
on  selected  tasks. 


dod  categorizes  simulations  into  the  following  three  classes  on  the  basis  of 
their  technical  and  physical  structure: 

•  Computer  simulation  models:  Armed  forces,  weapon  systems,  and  the 
interaction  between  them  are  primarily  represented  by  computer  code  and 
run  from  a  computer  keyboard.  Such  simulations  are  not  designed  to 
replicate  specific  weapon  systems.  Rather,  they  provide  a  view  of  the 
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Benefits  of  Simulation 
Training 


battle  on  a  map,  which  is  displayed  on  a  video  monitor  with  units  on  the 
map  shown  as  icons  (unit  symbols).  Computer  models  can  differ  greatly  in 
the  detail  they  contain  and  may  involve  some  human  interaction  while  the 
model  is  running.  War  games  foil  into  this  category. 

Weapon  system  simulators:  Individual  weapon  systems  are  modeled  and 
typically  controlled  by  a  human  operator.  Simulators  replicate  the 
performance  characteristics  and  appearance  of  a  weapon  system  to  help 
teach  individual  skills.  Aircraft  simulators  are  typical  examples.  Principal 
emphasis  is  now  on  networking  multiple  weapon  system  simulators 
together  in  an  interactive  format  to  create  a  large  scale  simulated  training 
environment  extending  beyond  the  training  of  individuals  and  crews  to 
include  multiple  crews  and  units.  One  example  is  the  networking  of  tank 
simulators  on  an  electronic  battlefield. 

Instrumented  exercises:  Actual  troops,  weapon  systems,  and  support 
systems  interact  in  as  real  an  environment  as  possible,  short  of  combat 
Electronic  instrumentation  is  used  to  collect,  analyze,  and  distribute  data 
on  force  elements.  Examples  include  the  Army’s  National  Training  Center 
in  California,  the  Air  Force’s  Red  Flag  in  Nevada,  and  the  Navy’s  Top  Gun 
Facility  in  California. 

Technology  developments  in  the  1990s  are  beginning  to  provide 
opportunities  to  connect  war  games  with  simulators  and  connect  both  of 
these  with  instrumented  training  ranges.1  Opportunities  will  exist  to 
connect  multiple  simulations  from  different  services  to  provide  joint 
training.  Training  experts  consider  these  developments  a  revolution  in 
military  training. 


Some  militaiy  officials  perceive  simulations  to  be  a  poor  alternative  to 
traditional  field  training  exercises.  However,  there  is  increasing 
recognition  that  while  traditional  field  training  exercises  are  important, 
they  can  have  significant  limitations — limitations  that  can  be  overcome  to 
a  certain  extent  through  the  use  of  simulations.  For  example,  traditional 
field  training  (such  as  Army  tank  maneuver  or  gunnery  training)  can  be 
limited  by  the  high  cost  of  fuel,  training  ammunition,  and  repair  parts;  lack 
of  space;  safety  and  environmental  concerns;  and  lack  of  time  required  to 
prepare  for  and  undertake  such  exercises.  A  prior  gao  report  addressed 
the  transition  of  reforger  (REtum  of  FORces  to  GERmany)  exercises  in 


'Unless  otherwise  stated,  the  use  of  the  term  simulation  in  this  report  refers  collectively  to  the  broad 
range  of  simulations  and  simulator  systems  used  for  training  purposes. 
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Germany  from  the  use  of  large-scale  field  exercises  to  the  increased  use  of 
simulation.2 

Computer  simulations  also  have  limitations,  but  they  provide  important 
training  opportunities  and  capabilities  not  always  feasible  in  traditional 
field  exercises.  Large-scale  field  exercises  that  emphasize  battle  planning 
and  command  and  control  of  forces  are  time-consuming  and  often 
produce  significant  downtime  for  lower  echelon  personnel.  Computer 
simulated  exercises  permit  more  concentrated  and  repetitive  training  for 
tactical  operations — such  as  tank  maneuvers  for  ground  forces — as  well  as 
command  and  control  operations  for  leaders  and  their  staffs.  Moreover, 
computer  simulated  training  provides  opportunities  for  individuals  and 
units  to  enhance  their  skill  levels  before  and  after  participating  in 
traditional  field  exercises.  Finally,  simulations  permit  training  that  is  not 
otherwise  practical  in  peacetime,  such  as  the  use  of  electronic  warfare 
that  might  be  restricted  due  to  safety  and  environmental  factors. 

Simulations  played  a  key  role  in  preparing  U.S.  forces  for  ground 
operations  in  Operation  Desert  Storm.3  As  a  result,  there  is  growing 
recognition  that  the  military  needs  to  use  this  technology  along  with 
traditional  field  training.  However,  as  noted  in  several  of  our  recent 
reports,  the  most  appropriate  mix  between  traditional  field  and  simulation 
training  has  not  been  established.4 


Beginning  in  1983,  the  Defense  Advanced  Research  Projects  Agency  and 
the  Army  jointly  pursued  a  project  to  develop  technology  for  constructing 
large-scale  computer  networks  to  create  a  battlefield  environment  where 
combat  teams  and  their  related  support  elements  could  operate  together 
on  a  regular  basis.  This  over  $200-million  effort  led  to  the  creation  of  a 
training  system  that  came  to  be  called  simnet  (SIMulation  NETworking). 
The  technology  associated  with  simnet  has  provided  the  foundation  for  the 
Army’s  current  efforts  to  acquire  an  advanced  network  simulation  system 
as  well  as  do D- wide  efforts  to  foster  use  of  this  technology. 


’Army  Training:  Computer  Simulations  Can  Improve  Command  Training  in  Large-Scale  Exercises 

(GA<W5IX&-91-6ri7kn  357TW 

’Operation  Desert  Storm:  War  Offers  Important  Insights  Into  Army  and  Marine  Corps  Training  Needs 
(GA(VN5rAM2-24<5;  Aug.  2571952}. 

4In  addition  to  reports  already  cited,  also  see  Army  Training:  Various  Factors  Create  Uncertainty  About 
Need  for  More  Land  (GACVNSIAD-91-103,  Apr.  22,1991). 
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In  simnet,  numerous  computer-driven  simulators  (configured  around  a 
mock-up  of  the  interior  of  an  armored  vehicle)  are  electronically  linked  to 
form  a  commc  '  combat  arena  or  battlefield.  Each  simulator  contains 
high-technology  electronics,  microcomputer  processing,  and  visual 
displays  with  three-dimensional  graphics  to  help  operators  believe  they 
are  in  actual  combat  Through  the  simulator’s  viewports,  which  are 
actually  small  video  monitors,  a  simnet  participant  sees  a  simplistic 
three-dimensional  landscape  and  other  simulated  vehicles  participating  in 
the  interactive  exercise.  From  the  driver’s  position,  an  operator  can 
maneuver  the  simulated  tank  across  the  terrain;  and  from  the  gunner’s 
station,  an  operator  can  locate  an  enemy  target  and  engage  the  main  gun 
to  hit  it  The  commander  can  simulate  the  rotation  of  the  tank’s  turret  to 
help  locate  targets  and  survey  the  battlefield.  A  sense  of  realism  is 
enhanced  through  auditory  cues  that  replicate  activities  expected  during  a 
battle.  They  include  replications  of  the  whine  of  the  1,500  horsepower 
turbine  engine  and  realistic  sounds  normally  associated  with  the  loading 
and  firing  of  the  main  gun. 

simnet  also  incorporates  some  battlefield  participants  that  are  generated 
by  the  computer.  These  participants  are  called  semi-automated  forces 
because  a  trainer  initiates  a  combat  task  from  a  computer  work  station, 
but  the  task  is  performed  automatically  according  to  specified  war-fighting 
doctrine  and  tactics  that  are  programmed  into  the  system.  Semi-automated 
forces  may  be  both  friendly  and  opposing  forces.  Other  functions  are 
initiated  from  computer  work  stations  to  depict  air  and  artillery  strikes 
and  provide  combat  service  support  by  dispatching  fuel,  maintenance,  and 
ammunition  trucks  when  needed. 

simnet  also  has  a  built-in  data  collection  and  analysis  system  that  records  a 
complete  history  of  a  simulated  battle,  including  orders  to  take  actions, 
the  resulting  actions,  and  the  impact  of  the  actions.  The  data  are  stored  so 
that  battle  participants  can  analyze  the  results  late,  in  after  action  reviews. 


simnet  was  first  demonstrated  in  May  1986,  at  the  Army’s  Armor  Center  in 
Fort  Knox,  Kentucky,  using  local  area  communications  networks.  A 
second  simnet  site  was  activated  in  March  1987  at  Grafenwohr  training 
center  in  Germany.  Long-haul  networks  to  link  simnet  units  operationally 
at  distant  sites  were  tested  in  1988  to  connect  Fort  Benning,  Georgia;  Fort 
Rucker,  Alabama;  Cambridge,  Massachusetts;  and  Washington,  D.C.,  with 
Fort  Knox,  Kentucky.  Ultimately,  236  simulators  were  acquired  and  placed 
in  11  sites  in  the  United  States  and  Germany;  they  have  remained  in 


Page  II 


GAO/NSIAD-9S-122  DOD  and  Army  SlmuUtioa  Traiaiag 


Chapter  1 
Introduction 


SIMNET  Limitations 


continual  use  as  training  devices.  Army  forces  used  simnet  to  prepare  for 
Operation  Desert  Storm. 

While  the  Defense  Advanced  Research  Projects  Agency  transitioned 
control  of  the  completed  simnet  training  system  to  the  Army,  it  expanded 
the  simnet  concept  to  other  cutting  edge  technologies  in  the  networking 
area.  In  one  area,  the  Defense  Advanced  Research  Projects  Agency  is 
pursuing  technology  to  connect  the  three  different  classes  of  simulations, 
including  those  from  different  services,  for  simultaneous  operations. 
Portions  of  this  technology  were  demonstrated  in  recent  exercises  in 
Korea  and  Germany. 

Success  of  the  simnet  program  has  led  to  the  formalization  of  the 
networking  protocols  as  an  industry  standard  known  as  Distributive 
Interactive  Simulation  (dis)  standards.  Formalization  of  dis  standards  is 
ongoing  through  government,  industry-,  and  academia-sponsored 
workshops  and  should  support  the  networking  of  multiple  simulations, 
including  those  from  various  services.  The  Army,  with  support  from  the 
other  services  and  DOD  organizations,  is  providing  much  of  the  leadership 
for  this  initiative.  The  dis  concept  has  been  expanded  into  several  other 
service-specific  initiatives.  For  example,  the  Air  Force’s  Armstrong 
Laboratory  has  developed  a  network  to  test  dis  applicability  to  fighter 
aircraft  The  Navy  is  developing  a  Tactical  Combat  Training  System  to 
provide  real-time  interactive  simulated  training  for  ships,  aircraft,  and 
submarines  while  they  are  at  sea. 


simnet  was  viewed  as  a  revolution  in  training.  It  enhanced  the  capability 
for  combat  crews  and  units  to  train  repetitively  in  a  fully  interactive 
combined  arms  environment  with  elements  of  the  military  forces 
represented  either  by  operator-controlled  simulators  or 
computer-generated,  semi-automated  forces.6  However,  simnet  had 
limitations.  Several  studies  were  undertaken  by  the  Army  to  assess 
simnet’s  training  benefit  These  studies,  most  of  which  produced 
nonquantitative  results,  showed  positive  training  benefits  but  noted  that 
some  tasks  were  being  trained  incorrectly  and  some  essential  tasks  were 
not  trained  at  all.  These  studies  identified  some  key  limitations  in  simnet, 
including  the  following: 


‘Combined  aims  training  refers  to  collective  training  that  includes  combat,  combat  support,  and 
combat-service  support  units.  Combat  units  comprise  Infantry  and  armor  forces.  Combat  support 
refers  to  fire  support  and  assistance  such  as  artillery  and  combat  engineering.  Combat-service  support 
refers  principal!}  to  logistics  and  administrative  support  such  as  supply  and  transportation. 
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•  Only  two  nuyor  ground  combat  vehicles  were  represented. 

•  Few  restrictions  were  imposed  on  vehicle  movements.  Terrain 
irregularities  such  as  ditches,  gullies,  or  stream  beds  were  not  adequately 
portrayed.  Trees  and  forests  did  not  present  obstacles  since  vehicles  could 
run  through  trees  without  losing  speed. 

•  Fields  of  view  from  the  video  monitors  were  smaller  than  in  actual 
vehicles,  and  the  gunnery  systems  were  unable  to  identify  targets  at 
realistic  distances. 

•  Commanders’  view  of  the  battlefield  was  constrained.  In  the  case  of  a  real 
tank,  commanders  could  peer  outside  through  a  popped  hatch  and  have  a 
360-degree  view. 

Additionally,  an  independent  assessment  of  the  simnet  software  by  a 
systems  engineering  company  found  many  technical  errors  in  computer 
code  written  to  depict  system  performance. 


Before  assuming  full  responsibility  for  simnet  in  1990,  the  Army  was 
already  addressing  simnet’s  deficiencies.  According  to  Army  officials, 
simply  improving  simnet  was  not  considered  a  viable  option  for  the 
following  reasons:  (1)  improvements  in  technology  were  making  the 
system  obsolete;  (2)  the  Army  did  not  have  adequate  documentation  of  the 
simnet  programming  code,  which  could  have  limited  competition  for  the 
successor  project;  and  (3)  the  Army  wanted  to  develop  a  family  of 
advanced  simulators  to  train  all  of  its  forces  under  an  umbrella  concept 
known  as  the  Combined  Arms  Tactical  Training  (catt)  system. 

The  Army  envisions  the  catt  system  as  a  combination  of  training  devices 
that  can  each  train  a  unique  segment  of  the  Army’s  forces  but  can  also 
operate  in  a  combined  arms  exercise.  Five  training  devices  are  currently 
included  in  the  catt  concept.  The  first  device  is  the  Close  Combat  Tactical 
Trainer  (ccrr)  for  armor  and  mechanized  infantry  forces.  Other  catt 
systems  that  may  be  developed  include  an  Aviation  Combined  Arms 
Tactical  Trainer,  an  Air  Defense  Combined  Arms  Tactical  Trainer,  an 
Engineering  Combined  Arms  Tactical  Trainer,  and  the  Closed  Loop 
Artillery  Simulation.  The  Army  is  also  considering  other  potential  systems 
for  intelligence  and  electronic  warfare. 

The  baseline  ccit  will  have  numerous  advanced  capabilities  over  simnet 
and  will  be  further  upgraded  after  it  is  fielded  through  a  process  of 
preplanned  product  improvements.  These  improvements  are  the  desired 
features  that  program  managers  perceived  as  too  expensive  to  include  in 
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the  basic  system  or  those  that  they  believed  could  not  be  technologically 
produced  at  the  time. 

The  Army  released  a  Request  for  Proposal  for  the  ccrrr  acquisition  on 
November  5, 1991,  and  on  November  30, 1992,  it  awarded  a  cost 
reimbursable  contract  in  excess  of  $409  million  for  the  engineering  and 
manufacturing  development  phase  of  the  acquisition  cycle.  Current 
milestone  dates  call  for  decisions  on  full-rate  production  to  be  made  in 
1997. 


Since  1976,  reports  by  various  dod  and  service  study  panels  have  reflected 
concerns  about  the  management  of  modeling  and  simulation  activities.  A 
1988  report  to  the  Secretary  of  Defense  from  a  Task  Force  of  the  Defense 
Science  Board  indicated  that  computer-based,  simulated  scenarios  offer 
the  only  practical  and  affordable  means  by  which  joint  commanders  and 
their  staffs  can  exercise  decision-making  skills,  test  war  plans,  and  train  as 
closely  coordinated  forces.  However,  the  Board  study  identified  problems 
with  the  management  of  modeling  and  simulation  activities.  The  study 
revealed,  for  example,  that  service  and  individual  joint  commanders  did 
not  coordinate  with  each  other  when  building  or  modifying  automated 
training  simulations,  and  therefore  built  duplicate  models.  A 1989  dod 
Inspector  General  report  on  dod  war-gaming  activities  stated  that  because 
the  Office  of  the  Secretary  of  Defense  and  the  Joint  Chiefs  of  Staff  lacked 
policies  and  guidance  on  war-gaming  activities,  dod  components  were 
purchasing  new  equipment,  developing  models  and  data  base  software, 
and  renovating  facilities  to  expand  joint  war-gaming  capabilities  without 
coordinating  these  efforts.  The  report  recommended  that  dod  designate  an 
office  of  primary  responsibility  within  the  Joint  Chiefs  of  Staff  to  establish 
policies  and  procedures  for  war  games  and  exercises. 

In  its  August  1990  Authorization  Report  for  Fiscal  Year  1991  Defense 
Authorizations,  the  House  Armed  Services  Committee  expressed  concerns 
about  the  adequacy  of  dod- wide  coordination  involving  computer 
simulation  technology.  The  Committee  recommended  the  establishment  of 
a  central  office  to  coordinate  the  technology  throughout  dod  and  raised 
questions  about  the  adequacy  of  ccrr  program  management  The 
Committee  was  especially  concerned  that  the  acquisition  strategy  be 
structured  to  ensure  that  normal  milestone  reviews  be  conducted  and  that 
operational  testing  receive  the  required  emphasis.  The  Committee  was 
also  concerned  about  system  interoperability;  the  integration  of  catt 
component  systems  with  other  training  activities,  from  a 
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cost-effectiveness  standpoint;  and  the  mix  of  catt  systems  needed.  Related 
concerns  had  also  been  expressed  in  an  Army  Audit  Agency  report® 


The  Chairman  of  the  House  Armed  Services  Committee,  requested  that  we 
review  dod’s  efforts  to  coordinate  simulation  technology’s  development 
and  use  agencywide.  The  Chairman  also  specifically  asked  us  to  review 
the  Army’s  plan  for  acquiring  catt  systems  to  ensure  that  it  provided  for 
(1)  high-level  management  oversight,  (2)  interoperability  with  other 
systems,  and  (3)  integration  of  the  system  with  more  traditional  forms  of 
training  for  greater  cost-effectiveness. 

To  assess  dod’s  role  in  providing  coordination  and  guidance  involving 
simulations,  we  interviewed  knowledgeable  officials  at  the  Defense 
Modeling  and  Simulation  Office  (dmso);  the  Operational  Plans  Directorate 
(J-7)  and  the  Force  Structure,  Resources,  and  Assessment  Directorate 
(J-8)  of  the  Joint  Staff;  the  Commander  of  the  Joint  Warfare  Center, 
Hurlburt  Field,  Florida;  the  Commanding  General  of  the  Marine  Corps 
Combat  Development  Center  in  Quantico,  Virginia,  and  other 
knowledgeable  Marine  Corps  training  officials;  and  modeling  and 
simulation  officers  from  the  Navy  and  the  Air  Force.  We  also  interviewed 
the  Executive  Director  of  the  National  Training  Systems  Association  in 
Arlington,  Virginia,  an  organization  representing  the  manufacturers  of 
training  simulations  and  other  training  systems.  We  reviewed  prior 
evaluations  by  Defense  Science  Boards  and  the  dod  Inspector  General, 
documentation  on  dmso’s  objectives  and  investment  initiatives,  and 
congressional  reports  on  the  subject  dmso’s  mission  encompasses  the  use 
of  modeling  and  simulation  in  the  areas  of  education,  training,  and  military 
operations;  research  and  development;  test  and  evaluation;  analysis;  and 
production  and  logistics;  however,  our  review  focused  on  training. 

To  determine  the  status  of  the  ccrr  acquisition  and  plans  for  other  catt 
components,  we  interviewed  officials  from  the  Army’s  Deputy  Chief  of 
Staff  for  Operations,  Arlington,  Virginia;  the  cctt  development  team 
members  at  the  Simulation,  Training,  and  Instrumentation  Command, 
Orlando,  Florida;  the  Training  and  Doctrine  Command’s  System  Manager 
for  catt,  Fort  Knox,  Kentucky;  the  Chief  of  the  Army  Model  and 
Simulation  Management  Office,  Crystal  City,  Virginia;  officials  at  the 
Combined  Arms  Command,  Fort  Leavenworth,  Kansas;  Training  and 
Doctrine  Command’s  Analysis  Center,  White  Sands  Missile  Range,  New 
Mexico;  and  Defense  Advanced  Research  Projects  Agency,  Arlington, 


’Networking  Training  Devices,  U.S.  Army  Audit  Agency  (SO  90-10,  Apr.  16, 1990). 
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Virginia.  We  reviewed  numerous  reports  on  simnet,  the  catt  Master  Plan, 
the  statement  of  work  and  the  contract  for  ccrr,  the  ccrr  Test  and 
Evaluation  Master  Plan,  preliminary  cctt  cost  information,  and 
independent  evaluations  by  the  Army  Audit  Agency  and  Army  Science 
Boards.  We  participated  in  a  demonstration  employing  an  existing  simnet 
training  simulator,  a  tank  driver  simulator,  and  an  Abrams  M1A1  tank  in 
Fort  Knox,  Kentucky.  We  conducted  our  review  in  accordance  with 
generally  accepted  government  auditing  standards  between 
November  1991  and  December  1992. 
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DOD-wide  guidance  and  coordination  of  simulation  technology  is  evolving 
under  dmso,  which  was  created  in  1991  within  the  Office  of  the  Secretary 
of  Defense,  dmso  is  responsible  for  providing  guidance  and  coordination 
and  fostering  simulation  networking  across  dod  and  the  military  services. 
dmso  is  making  progress  in  responding  to  its  mandate,  but  it  has  not  fully 
developed  (1)  a  master  plan  to  guide  simulation  development  and  use 
across  dod  or  (2)  measures  to  ensure  that  the  services  implement  adequate 
controls  over  the  development  and  modifications  of  models  and 
simulations.  The  Office  of  the  Secretary  of  Defense  has  not  yet  approved 
permanent  staff  for  dmso. 

The  Joint  Staff,  under  the  Chairman  of  the  Joint  Chiefs  of  Staff,  also  have 
roles  in  the  simulation  area,  with  a  particular  focus  on  supporting  regional 
commanders  in  chief  (cincs)  who  report  to  the  Secretary  of  Defense 
through  the  Chairman.1  While  the  missions  of  dmso  and  the  Joint  Staff  are 
complementary,  two  Joint  Staff  directorates  appear  to  have  overlapping 
responsibilities  and  interpret  them  differently.  As  a  result,  some 
duplication  of  effort  and  inefficiencies  exist.  However,  the  Joint  Staff  has 
begun  to  reassess  its  organizational  structure  and  staffing  and  may  clarify 
the  roles  and  responsibilities  of  these  two  directorates  in  the  simulation 
area. 


Efforts  to  Strengthen 
Coordination  and 
Management  of 
Simulation  Activities 


In  June  1991,  in  response  to  the  growing  concerns  about  the  inadequacy  of 
dod  coordination  and  the  lack  of  guidance  dealing  with  simulation 
activities,  the  Deputy  Secretary  of  Defense  established  dmso  under  the 
Director  of  Defense  Research  and  Engineering,  dmso  has  a  key  role  in 
strengthening  the  use  of  models  and  simulations  across  a  wide  range  of 
activities,  including  education,  training  and  military  operations;  test  and 
evaluation;  research  and  development;  analysis;  and  production  and 
logistics. 


'Reporting  to  the  Secretary  of  Defense  are  assorted  defense  agencies,  three  military  departments  (the 
Army,  the  Air  Force,  and  the  Navy),  and  the  CINCs  of  the  unified  and  specified  commands,  who  also 
operate  under  the  strategic  direction  of  the  Chairman,  Joint  Chiefs  of  Staff.  Unified  commands  are 
composed  of  forces  from  two  or  more  services  and  may  be  organised  on  a  geographical  or  regional 
basis,  such  as  Atlantic  Command,  European  Command,  Pacific  Command,  and  Central  Command. 
Specified  commands  are  organized  on  a  functional  basis  and  are  normally  made  up  of  forces  from  a 
single  service. 
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dmso  operates  under  the  guidance  of  a  high-level  executive  council, 
chaired  by  the  Director  of  Defense  Research  and  Engineering.2  dmso  has 
operated  with  a  core  staff  of  between  six  to  eight  persons  assigned  on  a 
temporary  basis  and  augmented  it  with  contractor  support  Much  of  dmso’s 
work  is  accomplished  through  the  use  of  working  groups  comprised  of 
representatives  drawn  from  Defense  agencies  and  the  military  services. 

In  fiscal  year  1993,  dmso  expects  to  issue  a  dod  directive  outlining 
requirements  for  a  DOD-wide  approach  for  developing  and  using  models 
and  simulations.  A  draft  of  the  directive  encourages  information  sharing, 
investments  in  common  technologies,  and  the  formulation  of  common 
standards  for  simulation  development  The  draft  also  requires  the  services 
and  dod  organizations  to  implement  management  systems  to  oversee  their 
modeling  and  simulation  activities  and  to  facilitate  interservice 
coordination.  Modeling  and  simulation  offices  have  been  or  are  being 
established  by  the  services.3 


dmso  is  seeking  to  foster  the  development  of  Defense-wide  standards,  data 
bases,  and  communications  capabilities  in  the  simulation  area.  In  this  way, 
the  office  intends  to  promote  simulation  system  interoperability  and 
stimulate  the  development  and  use  of  joint  models  and  simulations,  dmso 
pursues  these  goals  through  functional  and  technical  working  groups  and 
by  funding  various  simulation  projects  submitted  by  the  services,  Joint 
Staff,  and  Defense  agencies.  Illustrative  of  the  types  of  projects  funded  are 
the  following: 

•  dmso  supports  efforts  to  digitize  terrain  from  maps  and  then  input  that 
information  into  simulated  battle  scenarios.  Such  an  effort  can  foster  the 
development  of  common  terrain  data  bases  for  cross-service  use. 

•  dmso  supplements  ongoing  development  of  a  DOD-wide 
telecommunications  network  called  Defense  Simulation  Internet.  This 
project  was  initiated  by  the  Defense  Advanced  Research  Projects  Agency 
as  a  technology  demonstration  and  is  expected  to  facilitate  joint  training 
exercises  by  linking  various  war-gaming  and  simulation  centers. 


*The  DMSO  Director  serves  as  Executive  Secretary  to  the  Executive  Council  for  Modeling  and 
Simulation,  which  is  composed  of  senior-level  officials  from  the  services,  Joint  Staff,  Office  of  the 
Secretary  of  Defense,  and  other  defense  agencies. 

’The  scope  of  this  review  did  not  extend  to  examining  the  modeling  and  simulation  offices  of  the 
services  to  determine  their  effectiveness  in  bringing  about  a  coordinated  approach  to  simulation 
activities. 
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*  dmso  supports  technical  workshops  in  which  industry,  government,  and 
academia  work  to  prepare  standards  that  will  govern  the  development  of 
simulation  systems  throughout  dod. 

*  dmso  supports  the  development  of  a  computerized  clearing  house  of 
dod- wide  models  and  simulations  that  will  include  reports, 
correspondence,  and  software  descriptions.  Prototype  software  is 
expected  to  be  developed  within  the  next  several  months  and  data  to  be 
loaded  afterwards.  The  system  is  not  expected  to  be  fully  operational  until 
1994. 

*  dmso  funds  Navy  and  Marine  Corps  efforts  to  link  their  tactical  war-gaming 
systems  to  enhance  simulated  amphibious  operations,  dmso  has  also 
supported  the  Army  and  the  Air  Force’s  efforts  to  link  their  war-gaming 
models.  Likewise,  die  Navy  and  the  Air  Force  are  collaborating  on  a  Joint 
Air  Combat  Training  System. 


Need  for  Long-Range 
Master  Plan  and 
Controls  Over 
Simulation 
Development 


In  its  May  1992  Defense  Modeling  and  Simulation  Initiative  management 
plan,  dmso  outlined  its  vision,  goals,  objectives,  investment  strategy,  and 
future  activities.  In  this  plan,  dmso  recognizes  that  dod  does  not  have  a 
unified  statement  of  modeling  and  simulation  needs  and  deficiencies 
across  dod.  dmso  highlights  the  need  for  (1)  this  assessment,  (2)  the 
development  of  a  dod- wide  process  to  develop  individual  master  plans  for 
each  of  the  dod  components,  and  (3)  the  implementation  of  a  consolidated 
master  plan  that  draws  upon  the  individual  plans.  A  requirement  for 
individual  service  plans  is  included  in  the  draft  dod  directive  on  modeling 
and  simulation,  dmso  expects  to  complete  an  initial  draft  of  a  master  plan 
in  1993,  which  will  incorporate  some  of  the  services’  views  on  the  subject 


To  help  identify  needs  and  deficiencies  in  the  simulation  area,  dmso 
conducted  workshops  with  functional  users  (such  as  training  and  test  and 
evaluation)  from  each  of  the  services,  the  cincs,  defense  agencies,  and 
federally  funded  research  and  development  centers.  In  March  1993,  dmso 
published  the  results  of  the  workshops  and  envisions  producing  a  similar 
report  every  2  years.  Further,  dmso  expects  to  use  this  and  other 
information  to  guide  its  priorities  for  funding  projects  to  help  address  the 
deficiencies. 


In  its  modeling  and  simulation  initiatives  plan,  dmso  also  recognizes  the 
need  to  improve  controls  over  the  development  and  use  of  models  and 
simulations.  Two  areas  of  particular  concern  are  configuration 
management  and  the  verification,  validation,  and  accreditation  of  models 
Problems  with  configuration  management  result  when  organizations 
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DMSO  Staffing  and 
Funding 


modify  simulation  system  software  without  fully  documenting  changes. 
Lack  of  configuration  control  makes  it  difficult  to  ensure  system  reliability 
and  validity. 

Verification,  validation,  and  accreditation  are  needed  to  ensure  model 
reliability  and  consistency.  Verification  is  the  process  of  determining  that 
models  accurately  represent  the  developer’s  conceptual  description  and 
specifications.  Validation  determines  the  extent  to  which  models 
accurately  represent  the  real  world  from  the  perspective  of  the  intended 
users  of  die  simulation.  Accreditation  is  the  determination  that  the  model 
or  simulation  is  acceptable  for  its  intended  purposes. 

In  beginning  to  address  these  issues,  dmso  has  included  in  a  draft  dod 
directive  on  models  and  simulations  a  requirement  that  the  services  and 
other  defense  organizations  assign  responsibility  for  configuration  control 
to  a  specific  organization.4  dmso  also  has  included  in  this  draft  directive  a 
requirement  that  the  services  establish  a  verification,  validation,  and 
accreditation  program.  Further,  the  directive  assigns  responsibility  to  the 
services  for  accrediting  simulations  of  their  respective  forces  when  they 
are  included  in  models  developed  by  another  service. 


The  Office  of  the  Secretary  of  Defense  expected  dmso  to  operate  with  a 
small  staff  and  rely  on  interagency  groups  to  accomplish  its  mission.  The 
decision  by  the  Deputy  Secretary  of  Defense  to  create  dmso  carried  with  it 
an  expectation  that  dmso  staffing  would  be  found  within  existing 
personnel  levels — a  challenging  objective  in  the  face  of  pressures  to 
reduce  personnel  as  part  of  ongoing  dod  downsizing  efforts. 

dmso  lacks  permanent  personnel  billets  and  operates  with  a  temporary 
staff  that  has  fluctuated  in  size  from  approximately  six  to  eight 
government  personnel  on  loan  from  the  services.  This  staff  is  augmented 
by  up  to  eight  staff  years  of  contractor  support  The  dmso  Director, 
Technical  Director,  and  Budget  Officer  temporarily  occupy  some  of  the 
staff  positions.  However,  temporary  authorizations  for  the  director’s 
position  expired  in  August  1992,  and  the  authorization  for  the  Judget 
Officer  position  expired  at  the  end  of  calendar  year  1992.  Further,  dmso 
officials  have  been  or  are  in  the  process  of  being  reassigned.  Moreover, 
dmso  officials  have  sought  some  permanent  staffing  authority  for  their 
office,  but  as  of  April  1993,  they  had  not  obtained  formal  approval.  The 


*This  directive,  currently  being  reviewed  by  the  services,  defense  agencies,  and  Joint  Staff 
organizations,  b  expected  to  be  issued  this  fiscal  year. 
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Table  2.1:  DMSO  Funding  for  Fiscal 
Years  1991-93 


Overlapping  Roles 
Between  Joint  Staff 
Directorates 


officials  seek,  for  example,  to  fill  a  civilian  senior  executive  position  to 
provide  a  permanent  director  for  the  office.  The  absence  of  permanent 
staffing  could  create  gaps  in  institutional  knowledge  and  hamper  mission 
continuity. 

Although  the  Congress  strongly  supported  dmso  by  creating  and  initially 
funding  it,  internal  dod  budget  reviews  have  created  uncertainty  about 
future  funding  prospects.  In  fiscal  year  1991,  Congress  appropriated 
$75  million  for  dmso,  but  $50  million  of  that  money  was  included  in  a  dod 
budget  recision.  The  following  year,  Congress  appropriated  $50  million, 
approximately  $20  million  of  which  was  lost  through  the  dod  budget 
recision  process,  dmso  has  a  fiscal  year  1993  congressional  appropriation 
of  $70  million.  Table  2.1  summarizes  dmso  funding  from  fiscal  year  1991 
through  fiscal  year  1993. 


Dollars  in  millions 

Fiscal  year 

Research  and 
development 

Procurement 

Total 

appropriated 

Total  after  DOD 
budget 
reeisions 

1991 

$15 

$60 

$75 

$25 

1992 

40 

10 

50 

30 

1993 

60 

10 

70 

70s 

•No  recision  action  as  of  February  1993. 

The  missions  of  dmso  and  the  Joint  Staff  are  complementary,  with  both 
addressing  some  aspects  of  joint  training.  In  contrast  with  dmso,  which  is 
filling  a  broader  mandate  and  more  of  an  integrating  role  across  the 
military  services,  the  efforts  of  the  Joint  Staff  focus  on  joint  operational 
matters  involving  cincs.  However,  within  the  Joint  Staff,  two  directorates 
that  we  reviewed  have  unnecessarily  overlapping  responsibilities  in 
providing  simulation  assistance  to  the  cincs,  and  each  brings  a  different 
perspective  on  how  to  provide  such  support 

According  to  Joint  Administrative  Publication  1.1,  August  1992,  several 
Joint  Staff  directorates  have  responsibilities  for  various  aspects  of 
modeling  and  simulation.  Two  of  these  directorates,  J-7  and  J-8,  have 
assigned  responsibilities  for  supporting  cincs’  training  activities  with 
computer  simulations.  For  example,  J-7  oversees  operations  of  the  Joint 
Warfare  Center,  a  computer  war-gaming  center  located  at  Hurlburt  Field, 
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Florida.6  This  center,  which  until  the  late  1980s,  had  operated  under  the 
now  defunct  Readiness  Command  and  subsequently  under  Forces 
Command,  was  instrumental  in  developing  simulations  and  supporting 
joint  exercises  conducted  by  the  unified  and  specified  cincs.6  Further,  in 
March  1992,  J-7  established  a  Joint  Simulation  and  Interoperability 
Division  (jsid)  and  tasked  it  to  coordinate  modeling  and  simulation 
activities  involving  the  cincs.  Specifically,  the  division’s  mission  is  to 
promote  the  application  of  models  and  simulations  in  joint  operational 
planning  and  execution,  education,  training,  exercises,  operations 
requirements,  joint  test  and  evaluation,  and  doctrine  development  and 
evaluation.  The  division  is  expected  to  monitor  service  modeling  and 
simulation  programs  and  development  activities  to  encourage  joint 
applications  within  these  areas. 

At  the  same  time,  J-8  is  tasked  with  developing  joint  modeling  and 
simulation  capabilities  as  necessary  to  support  assigned  missions, 
including  conducting  joint  war  games  and  interagency  politico-military 
seminars  and  simulations.  J-8’s  assigned  missions  were  not  changed  with 
the  creation  of  jsid.  J-8’s  technical  director  said  his  organization  has  long 
been  responsible  for  providing  or  arranging  for  simulation  support  to  the 
cincs  across  a  broad  spectrum  of  functions,  including  training,  education, 
operational  planning,  and  analysis.  The  technical  director  said  this  support 
is  provided  by  his  staff  of  over  80  operations  research  analysts  and 
through  agreements  with  the  services’  simulation  experts,  such  as  the 
Army’s  National  Simulation  Center.  J-8  is  also  the  focal  point  for  the  Joint 
Staff's  Modem  Aids  to  Planning  Program  through  which  cincs  obtained 
their  own  simulation  hardware  and  software  capabilities. 

Officials  in  each  of  the  Joint  Staff  directorates  have  voiced  differing 
perspectives  on  how  simulation  support  for  cincs  should  be  provided.  An 
area  of  long-standing  concern  to  Congress  has  been  the  lack  of  a  clear 
assessment  of  cinc  simulation  requirements.  Both  of  the  Joint  Staff 
directorates  claim  some  responsibility  for  completing  this  assessment  J-8 
has  surveyed  cincs  to  identify  their  current  simulation  capabilities  and 
expected  requirements  for  training,  education,  analysis,  and  operations 
support.  The  survey  was  undertaken  to  establish  a  baseline  capability  from 


‘The  Joint  Warfare  Center  l a  the  only  war-gaming  center  funded  and  operated  under  the  Joint  Staff. 
Other  war-gaming  centers  are  funded  by  an  Individual  service,  such  as  the  Army’s  National  Simulation 
Center  In  Fort  Leavenworth,  Kansas,  or  by  multiple  services,  such  as  the  Army/Air  Force-funded 
Warrior  Preparation  Center  in  Germany.  A 1989  DOD  Inspector  General  Report  recommended  that  the 
Joint  Warfare  Center  be  disestablished  because  it  duplicated  the  simulation  capabilities  of  other 
services. 

*The  Joint  Warfare  Center’s  mission  was  revised  in  May  1981  to  delete  any  continuing  role  in  the 
development  of  models  and  simulations.  It  now  focuses  primarily  on  supporting  CINC  joint  exercises 
and  training  programs. 
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which  an  investment  strategy  and  master  plan  could  be  developed.  A 
report  summarizing  the  questionnaire  results  was  issued  in  October  1992. 
This  report  is  the  first  of  a  three-volume  report  A  second  volume  is 
expected  to  discuss  a  strategy  for  meeting  the  identified  needs,  and  a  third 
volume  is  expected  to  outline  an  investment  plan.  The  last  two  volumes 
are  expected  to  be  completed  in  1993. 

The  jsid  Director  told  us  that  the  J-8  survey  is  heavily  weighted  toward 
analysis  issues  and  hardware  needs  but  does  not  fully  address  operational 
and  training  needs  of  the  cincs.  Our  review  of  the  J-8  questionnaire 
summary  indicated  that  ciNC-identified  simulation  needs  covered  a  broad 
spectrum  of  areas,  including  training,  operations,  education,  and  analysis. 

In  a  separate  effort,  the  J-Ts  Joint  Warfare  Center  has  begun  soliciting 
information  from  the  cincs  on  the  simulation  requirements  needed  to 
participate  in  some  advanced  technology  demonstrations,  jsid  officials 
said  this  inquiry  responds  to  recommendations  anticipated  from  a  1992 
Defense  Science  Board  Task  Force. 

When  first  established,  jsid  was  concerned  about  duplication  among  the 
cincs  and  had  considered  a  process  to  have  cinc  training  requirements 
funneled  through  it  before  steps  could  be  taken  to  develop  new 
simulations.  The  intent  of  such  a  proposed  process  was  to  ensure  that 
requirements  identified  by  one  cinc  would  be  reviewed  by  other  cincs  to 
determine  whether  they  had  similar  needs.  Once  the  needs  were 
determined,  jsid  had  planned  to  work  with  the  cincs  and  the  technical 
experts  in  J-8  and  the  Joint  Warfare  Center  to  identify  the  best  way  to  meet 
all  cincs’  needs. 

The  J-8  technical  director  told  us  that  his  division  currently  compares 
requirements  of  the  various  cincs  to  minimize  duplication  where  possible 
in  meeting  the  needs  and  that  he  believes  another  process  is  not  needed. 
However,  some  cincs  have  expanded  their  own  simulation  capabilities 
independent  of  Joint  Staff  support  In  some  cases,  cincs  obtain  assistance 
from  other  simulation  centers  operated  by  the  Army  and  Air  Force.  For 
example,  during  the  recent  exercises  in  Korea,  staff  from  the  Army’s 
National  Simulation  Center,  rather  than  the  Joint  Warfare  Center, 
supported  the  cinc.  Thus,  at  the  present  time,  simulation  support  for  cinc 
activities  comes  from  multiple  sources  within  and  external  to  the  Joint 
Staff. 
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; 


Conclusions 


According  to  the  jsid  Director,  a  central  review  process  in  jsid  was  not 
implemented,  but  the  director  believes  more  effort  is  needed  to  better 
integrate  cinc  training  requirements  for  simulations  with  other  functions. 
To  this  end,  the  Joint  Staff  initiated  a  study  to  assess  whether  a 
consolidated  joint  training,  doctrine,  and  simulation  center  is  needed.  The 
first  part,  completed  in  February  1993,  indicated  a  need  to  bring  trainers 
together  with  developers  of  military  doctrine  and  to  bring  both  of  these 
functions  together  with  the  developers  of  training  tools,  like  simulations. 
The  second  part  of  the  study,  expected  to  be  completed  in  July  1993,  will 
determine  the  most  appropriate  structure  for  the  center.  Among  other 
structures  being  considered  is  the  consolidation  of  the  existing  Joint 
Warfare  Center,  the  Joint  Doctrine  Center,  and  jsid. 

In  addition,  two  other  efforts  are  underway  that  might  eliminate  some  of 
the  overlap.  The  Vice  Director  of  the  Joint  Staff  recently  initiated  a 
2-month  study  of  the  organization  and  staffing  of  the  Joint  Staff.  Because 
of  downsizing  in  DOD,  this  study  is  intended  to  identify  where  additional 
staff  can  be  eliminated,  and  one  area  of  interest  will  be  overlapping 
functions.  Finally,  the  Chairman,  in  response  to  legislative  requirements,  is 
conducting  a  comprehensive  study  of  the  roles  and  missions  of  the  armed 
forces.  One  area  being  considered  is  an  integrated  training  and  test  and 
evaluation  structure  that  will  electronically  link  the  services’  training  and 
test  ranges  in  six  western  states.  This  structure  is  expected  to  provide  a 
land,  airspace,  sea  area,  and  offshore  supersonic  operating  domain  to 
accommodate  a  large  portion  of  joint  training  and  test  and  evaluation 
needs. 


A  dod- wide  structure  for  providing  oversight  and  guidance  regarding 
simulation  technology  is  in  place,  and  progress  is  underway  in  key  areas. 
dmso  has  provided  an  important  focal  point  for  the  simulation  initiative  but 
has  yet  to  complete  a  master  plan  for  meeting  identified  needs  and 
deficiencies  or  ensure  adequate  controls  over  simulation  development 
Continued  progress  could  be  adversely  affected  by  a  lack  of  permanent 
staffing  for  the  office.  Despite  progress  made  at  dmso,  much  remains  to  be 
done  to  sort  out  the  roles  and  responsibilities  of  the  Joint  Staff, 
particularly  between  J-7  and  J-8  in  the  simulation  area  Three  studies  are 
underway  or  anticipated  that  may  help  resolve  the  inefficient  use  of 
resources  that  now  exists  and  better  address  simulation  support  for  the 
CINCS. 
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Recommendations 


Agency  Comments 
and  Our  Evaluation 


We  recommend  that  the  Secretary  of  Defense 

•  ensure  that  dmso  is  properly  staffed  to  carry  out  its  assigned 
responsibilities  and 

•  eliminate  unnecessary  inefficiencies  within  the  two  Joint  Staff  directorates 
involved  in  simulations  a'ld  more  clearly  delineate  each  directorate’s  roles 
and  responsibilities  in  this  area 


In  commenting  on  a  draft  of  this  report,  dod  concurred  with  our 
recommendation  to  properly  staff  dmso  and  partially  concurred  with  our 
recommendation  to  eliminate  duplication  within  the  Joint  Staff,  dod  said 
that  responsibilities  for  modeling  and  simulation  functions  were  shared  by 
several  directorates,  not  just  those  included  in  this  report  dod  said  that 
this  does  not  constitute  unnecessary  duplication  or  overlap  of  effort  dod’s 
comments  are  presented  in  their  entirety  in  appendix  II. 

Since  we  found  overlapping  functions  betv.  jen  two  joint  staff  directorates, 
especially  with  respect  to  identifying  modeling  and  simulation 
requirements  for  ClNCs,  which  resulted  in  inefficient  use  of  resources, 
tod’s  acknowledgement  that  even  more  joint  staff  directorates  have 
modeling  and  simulation  responsibilities  does  not  dispel  our  concern,  but 
rather  serves  to  reinforce  it.  Moreover,  for  dod  to  express  the  opinion  that 
there  is  no  unnecessary  duplication  before  its  own  studies  of  the  issue  are 
completed  seems  to  undermine  the  purpose  of  the  studies. 
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Responding  to  congressional  concerns  about  the  management  of  ccrr  and 
catt  acquisitions,  the  Army  formulated  a  plan  to  ensure  that  these  systems 
would  have  high-level  oversight  comparable  to  that  provided  for  nuqor 
weapon  system  acquisitions.  The  Army  also  strengthened  its  management 
structure  for  acquiring  simulations.  The  revised  structure  is  intended  to 
elevate  acquisition  decisions  to  a  higher  level  and  to  ensure  more  direct 
involvement  of  the  Army  Acquisition  Executive — a  role  filled  by  the 
Assistant  Secretary  of  the  Army  for  Research,  Dev  elopment,  and 
Acquisition.  These  actions  affect  the  acquisition  of  ccrr  and  other 
simulation  systems.  In  addition,  the  Army  also  has  taken  steps  to  ensure 
interoperability  between  ccrr  and  follow-on  catt  systems,  and  other 
simulations. 


CCTT  and  Other 
CATT  Components  to 
Be  Acquired  as  Mzyor 
Systems 


In  November  1991,  the  Army  developed  a  master  plan  to  outline  the 
management  structure  for  acquiring  the  cctt  and  other  simulations  in  the 
catt  system.  The  master  plan  stated  that  the  catt  system  will  be  a  series  of 
unique  training  devices  for  each  of  the  Army’s  combat  forces  (armor, 
aviation,  air  defense,  and  artillery,  for  example).  While  these  training 
devices — called  catt  components — will  be  developed  with  the  capability 
to  electronically  link  together,  each  device,  according  to  Army  officials,  is 
a  complete  stand-alone  training  system  that  can  be  operated 
independently  of  other  components.  This  approach  seems  consistent  with 
the  dis  concept,  which  is  based  on  systematically  linking  numerous 
operational  subcomponents  to  create  a  larger  system. 


Because  of  its  estimated  cost  in  excess  of  $1  billion  and  high  visibility,  the 
ccrr  was  designated  as  a  category  II  major  defense  acquisition  program.1 
Such  a  designation  requires  the  Army  Acquisition  Executive  to  approve 
kev  acquisition  decisions  following  milestone  reviews  by  the  Army 
Systems  Acquisition  Review  Council.2  Analytical  evidence  must  be 
presented  at  these  milestone  reviews  to  justify  continuing  the  program  and 
ensure  that  the  program  meets  specific  requirements  in  the  test  and 
evaluation  area.  Appendix  I  contains  an  overview  of  milestone  review 
requirements,  and  table  3.1  shows  cctt’s  milestones  and  other  key  dates. 


'C«tegory  n  programs  have  research,  development,  test,  and  evaluation  expenditures  of  $115  million  or 
more  or  a  procurement  expenditure  of  *640  million  or  more  (in  1990  dollars).  These  programs  must  be 
approved  by  an  official  no  lower  than  the  Acquisition  Executive  for  the  specific  DOD  component 

*The  members  of  this  council  include  top  managers  from  a  broad  spectrum  of  Army  organisations, 
such  as  the  Assistant  Secretaries  for  Financial  Management  and  Research,  Development,  and 
Acquisition;  various  Chiefs  of  Staff  and  their  deputies;  officials  from  the  training  and  testing 
communities;  and  others. 
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Table  3.1:  CCTT  Milestone*  as  of 
December  1992 


Since  much  of  0071*8  conceptual  development  had  been  accomplished 
under  simnet,  two  of  the  normal  reviews  were  combined. 


Jl  l  [  l  ^  ^  — g 

Milestone 

Date 

Activity 

0 

Under  SIMNET 

Concept  explored  and  demonstrated 

I/ll 

June  1991 

Preliminary  approval  of  engineering  and 
manufacturing  development 

I/ll 

Oct.  1991 

Final  approval  of  development 

— 

Nov.  1992 

Contract  award 

— 

Sept.  1994 

Deadline  for  software  critical  design 
review 

— 

Oct.  1995 

Deadline  for  detailed  test  and 
evaluation  plan 

— 

May  1996 

Begin  fielding  limited-capability  units 

— 

Dec. 1996 

Begin  initial  operational  test  and 
evaluation 

Ill  A 

Apr.  1997 

Decide  low-rate  of  initial  production 

IIIB 

Dec.  1997 

Decide  full-rate  of  production 

— 

Aug.  1998 

First  system  delivered 

IV 

Not  determined 

Upgrades  and  improvements 

To  expedite  the  production  and  fielding  of  ccrr,  the  Army  currently  plans 
to  divide  its  milestone  m  decision  into  two  parts — the  first  for  a  low  rate 
of  initial  production  and  the  second  for  full  production.  Army  officials 
hope  to  have  sufficient  testing  data  by  the  milestone  in  A  decision  point  to 
authorize  a  low  rate  of  production  while  awaiting  final  compilation  and 
printing  of  test  results  for  the  milestone  in  B  full-production  review. 

Like  cctt,  each  follow-on  catt  component  is  expected  to  be  acquired  as  a 
m^jor,  separate,  stand-alone  system  with  funding  budgeted  as  a  separate 
program  line  item,  cctt  funding  through  fiscal  year  1993  has  been  included 
in  the  Army’s  overall  funding  line  for  training  devices.  Separate  line  item 
funding  for  cctt  is  planned  for  fiscal  year  1994  and  beyond  and  for  each 
subsequent  catt  component 

The  Army  awarded  a  contract  to  the  International  Business  Machines 
Corporation  on  November  30, 1992,  to  begin  engineering  and 
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manufacturing  design  of  the  cctt.3  Hie  contract  is  valued  over  $409  million 
and  covers  the  development  of  49  cctt  prototypes  to  be  used  for  testing 
and  the  production  of  68  limited-capability  units. 


New  Command  to 
Manage  the 
Acquisition  of 
Simulations 


Until  recently,  the  Army  Material  Command’s  Project  Manager  for  Training 
Devices  in  Orlando,  Florida,  was  responsible  for  helping  Army 
components  define  user  requirements  for  a  wide  range  of  training  devices 
and  simulations  and  for  managing  related  acquisitions.4  In  the  case  of  cctt, 
the  Project  Manager  for  Training  Devices  worked  with  the  Army’s  Armor 
School  at  Fort  Knox,  Kentucky,  to  develop  system  requirements.  In 
August  1992,  the  Army  created  a  new  nuyor  subordinate  command  known 
as  the  Simulation,  Training,  and  Instrumentation  Command  to  centralize 
oversight  of  technology  development  and  life-cycle  support  for  training 
devices;  Instrumentation,  Targets,  and  Threat  Simulations;  and  other  Army 
battlefield  simulations. 

This  new  structure  is  intended  to  assign  higher  level  attention  to 
day-to-day  program  management  responsibility  over  cctt  and  is  designed 
to  provide  more  direct  lines  of  communication  with  the  Army’s  Acquisition 
Executive  in  m^jor  decisions  affecting  catt  components.  The  Army  also 
appointed  a  separate  Project  Manager  for  catt  to  manage  the  development 
and  fielding  of  cctt  and  other  catt  systems. 


Other  CATT 
Components  to 
Follow  Same  Process 
as  CCTT 


Future  catt  component  acquisitions  are  expected  to  follow  the  same 
acquisition  process  as  cctt.  Acquisition  milestones  for  other  catt  systems 
are  undecided  for  now;  however,  requirements  documentation  and 
reviews  are  further  along  for  the  aviation  trainer  than  for  the  other 
systems.  We  were  told  that  in  an  era  of  declining  defense  resources,  cctt  is 
currently  a  high  priority  among  senior  Army  leaders.  Less  clear  is  the 
degree  of  funding  support  for  other  catt  systems  in  relation  to  competing 
acquisition  systems. 


*The  Army’s  contract  award  for  CCTT  was  delayed  2  months  beyond  the  expected  award  date  because 
of  concern  on  the  part  of  Army  officials,  particularly  the  Army's  Acquisition  Executive,  about  cost 
realism  of  individual,  competing  firms.  After  discussions  between  the  Acquisition  Executive  and 
representatives  of  the  competing  firms,  the  firms  raised  their  cost  proposals  in  varying  amounts  so  that 
they  were  much  closer  to  the  Army's  valuation  of  the  likely  costs. 

*This  Project  Manager  is  colocated  with  the  Naval  Training  Systems  Center,  which  provides  the  Army 
with  technical  contracting  assistance  in  acquiring  training  devices  and  simulations. 
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The  Army  has  acted  to  ensure  high-level  executive  oversight  of  ccrr  and 
involvement  in  key  milestone  decisions.  It  has  also  reorganized  and 
elevated  the  management  of  the  ccrr  program.  Further,  the  Army  has 
included  requirements  in  the  catt  master  plan  and  the  cctt  specifications 
to  ensure  interoperability  among  simulation  systems. 
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The  Army  is  integrating  simulation  technology  with  traditional  field 
training  programs;  however,  the  question  of  what  is  the  most  cost-effective 
mix  of  field  and  simulation  training  has  not  been  quantified.  Until  the 
Army  completes  operational  testing  and  cost-effectiveness  assessments 
(expected  in  1996),  the  most  efficient  and  effective  mix  of  ccrr  and 
traditional  training  will  not  be  known.  Pressures  from  some  Army 
commanders  to  accelerate  fielding  of  cctt  have  led  to  plans  to  field 
68  limited-capability  units  in  Germany  and  the  United  States  before  the 
completion  of  operational  testing.  Differences  of  opinion  exist  over  how 
quickly  the  full  system  can  or  should  be  developed  and  fielded.  However, 
our  review  indicated  that  shortening  the  acquisition  process  to  accelerate 
system  development  and  fielding  could  have  several  important  drawbacks. 


Lack  of  Validated  Data  1116  Army  ^  develoP“)8  a  combine<i  arms  training  strategy  to  help  manage 

its  training  resources  in  a  more  integrated  manner.  When  completed,  this 
Oil  MlX  01  simulations  strategy  is  intended  to  establish  a  definitive  relationship  between  the  mix 

and  Field  Training  of  field  and  simulation  training.  However,  the  strategy  is  still  evolving,  and 

it  is  not  clear  when  definitive  data  will  be  available  to  provide  precise 
answers  to  questions  about  the  most  appropriate  mix  of  field  and 
simulation  training  or  the  best  combination  of  catt  systems.  This  issue  is 
further  affected  by  decisions  to  reduce  the  planned  sets  of  ccrr  to  be 
acquired  and  the  echelon  on  which  training  will  be  focused. 


Combined  Arms  Training 
Strategy  to  Help  Manage 
Training  Resources 


Since  1988,  the  Army  has  been  working  toward  developing  a  strategy  to 
better  identify  the  training  requirements  of  its  total  force  (armor,  infantry, 
aviation,  etc.);  a  strategy  that  will  enable  it  to  manage  all  training  in  an 
integrated  manner.  The  Army  intends  for  this  strategy  to  provide  guidance 
on  how  to  efficiently  mix  simulation  capabilities  with  more  traditional 
field  training — simply  stated,  the  “right  mix."  By  showing  training  events  to 
be  conducted,  frequencies,  and  resources  needed  to  conduct  those  events, 
the  Army  believes  that  a  combined  arms  training  strategy  could  help 
measure  the  relative  value  of  each  resource  needed  to  train  to  standard. 
Because  the  standard  remains  constant,  the  strategy  is  expected  to  clarify 
where  changes  in  resources  will  impact  training  and  allow  decisions  on 
trade-offs. 


At  the  heart  of  the  Combined  Arms  Training  Strategy  is  a  series  of 
individual  training  strategies  to  be  developed  by  the  Army’s  training 
centers  and  schools.  The  strategies  are  expected  to  identify  the  best  mix  of 
field  exercises  and  simulations  and  ensure  that  the  effectiveness  of 
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simulators  is  determined  before  critical  acquisition  decisions  are  made.  As 
of  January  1993,  only  coordinating  drafts  of  the  strategy  from  various 
doctrinal  schools  had  been  prepared.  The  Armor  Strategy,  which  relates  to 
training  involving  the  cctt,  contains  only  tentative  data  (that  is  not 
validated  by  user  testing)  regarding  the  right  mix  between  cctt  and  field 
training. 


Uncertainties  Over  Use  of 
CCTT  Simulators  and  How 
Best  to  Integrate  Them 
With  Field  Training 


The  Army  has  estimated  that  a  total  of  546  cctt  units  will  be  needed  to 
train  the  active  and  reserve  armor  units  at  the  echelons  of  company  level 
and  below.1  Table  4.1  summarizes  the  quantities  and  the  delivery  schedule 
for  the  cctt.  In  determining  the  desired  quantities  of  cctt  units  and 
follow-on  catt  systems,  the  Army  considered  the  expected  future  force 
structure,  the  draft  armor  training  strategy,  and  information  from  a  1989 
review  of  annual  training  plans  for  armor  and  mechanized  infantry  units. 


Table  4.1:  Planned  Quantities  and 
Deliveries  of  CCTT  Units 


Number  of 

Type  of 

Initial 

unite 

unit 

delivery  dates 

68 

Limited  capability  in  fixed  sites 

May  1996 

38 

Fixed-site  testing  prototypes 

Oct.  1996 

11 

Mobile  testing  prototypes 

Jan.  1997 

326 

Full  capability  in  fixed  sites 

Oct.  1998 

103 

Full  capability  mobiles 

July  1999 

546 

(432  In  fixed  eltes  and  114  mobiles) 

Training  staff  determined  that  approximately  20  days  a  year  could  be 
devoted  to  training  with  cctt,  providing  trade-offs  were  made  with  other 
scheduled  activities.  Thus,  the  cctt  acquisition  plan  incorporates  the  idea 
of  active  duty  company-sized  units  spending  20  days  per  year  training  with 
cctt  and  23  days  training  in  the  field.  The  plan  assumes  that  reserve  units 
will  train  using  mobile  cctt  units  once  or  twice  a  year  for  a  total  of  4  to  6 
days  per  year.  Army  officials  recognize  that  their  numbers  are  tentative 
and  must  be  validated  through  user  testing.  Key  questions  to  be  answered 
through  user  testing  include  the  following:  (1)  How  long  and  at  what 
intervals  should  units  train  on  ccrr?  (2)  To  what  extent  should  cctt  be 
used  as  a  prerequisite  to  certain  field  exercises?  and  (3)  To  what  extent 
should  cctt  be  used  to  reinforce  or  sustain  skills  previously  acquired? 


'The  term  ‘echelon*  refers  to  a  level  of  command  Platoon,  company,  battalion,  brigade,  and  division 
are  aucceedingly  higher  echelons  of  command 
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In  developing  its  acquisition  plan,  the  Army  decided  to  configure  over 
20  percent  of  total  cctt  units  in  mobile  sites  although  it  did  not  have 
experience  with  mobile  units  to  provide  a  more  informed  basis  for  its 
decision.  Mobile  simnet  units  were  not  placed  in  the  inventory  until 
July  1091,  and  an  evaluation  of  mobile  site  experience  is  not  anticipated 
until  early  1994. 

The  Army  estimated  that  training  in  other  catt  components  should  be 
available  to  active  duty  company  size  units  between  12  and  25  days  per 
year  and  reserve  units  should  have  access  between  2  and  4  days  every 
year.  The  Army  estimated  quantities  for  catt  systems  based  on  these 
frequencies,  and  like  cctt,  these  estimates  are  theoretical.  The  frequencies 
and  quantities  were  not  based  on  validated  assessments  of  the  training 
effectiveness  of  specific  simulations  because  none  of  the  catt  systems 
currently  exists.  The  Army  recognizes  that  the  most  appropriate  mix 
between  field  training  days  and  simulation  time  for  other  catt  systems  is 
still  subject  to  validation. 


Changes  in  Training  Focus  During  the  early  stages  of  developing  the  catt  concept,  the  Army 
and  the  Combination  of  anticipated  that  the  simulations  would  provide  tactical  training  for 

CATT  Systems  battalion  levels  and  below  by  networking  sites  from  remote  locations 

using  long-haul  networks.  It  initially  planned  to  acquire  up  to  2,754  units. 
Since  that  time,  the  Army  has  lowered  the  training  focus  to  company  and 
platoon  level  using  local  area  networks  and  reduced  the  number  of  cctt 
units  to  546.  (Battalion  sizes  and  long-haul  networks  are  listed  as  potential 
system  upgrades  in  the  future.)  Part  of  the  rationale  for  this  change  is  cost 
related,  but  it  also  includes  a  realization  that  the  amount  of  networking 
needed  is  unknown  since  unanswered  questions  exist  about  how  often  the 
various  elements  of  the  combined  forces  above  the  company  level  would 
or  should  train  together. 

However,  battalion-level  tactical  training  needs  can  be  partially  met  using 
the  cctt’s  semi-automated  forces  workstations  to  simulate  additional 
forces — friendly  or  opposing.  Further,  at  echelons  of  battalion  and  above, 
where  there  is  increasing  emphasis  on  command  and  control,  much 
training  can  be  provided  through  other  computer-simulated,  war-gaming 
exercises.  Additionally,  dod  is  working  on  capabilities  that  would  enable 
these  war-gaming  simulations  to  be  linked  with  exercises  conducted  in 
systems  such  as  cctt  and  those  from  other  services  and  with  field  training. 
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The  Army  System  Acquisition  Review  Council  tentatively  approved  ccrr’s 
engineering  and  manufacturing  development  phase  at  the  June  1991 
combined  milestone  I/n  review,  in  part,  on  the  basis  of  a  preliminary  cost 
and  training  effectiveness  analysis.  Army  officials  recognized  that  the  data 
used  in  this  analysis  were  “soft”  and  would  need  to  be  validated.  The 
preliminary  analysis  concluded  that  cctt  could  be  a  cost-effective  addition 
to  the  Army's  training  program  but  recommended  that  savings  associated 
with  ccrr  be  quantitatively  validated  based  on  the  results  of  user  testing. 
The  review  council’s  approval  also  stated  that  potential  reductions  in 
training  resources  due  to  the  ccit  must  be  validated  by  test  results  and  not 
based  on  artificial  numbers. 

The  preliminary  cost  and  training  effectiveness  analysis  was  performed  by 
analysts  from  the  Training  and  Doctrine  Analysis  Command  at  White 
Sands  Missile  Range,  New  Mexico,  in  May  1991.  The  analysis  was  based 
upon  existing  assessments  of  benefits  associated  with  ccrr’s  predecessor 
system,  estimated  training  data  from  the  Armor  School,  and  total  life-cycle 
costs  then  estimated  at  $1.2  billion  for  546  units  with  65  percent  of  the  cost 
attributable  to  the  active  component  and  35  percent  to  the  reserves.2  The 
analysis  concluded  that  the  active  component  portion  of  the  costs  could 
be  fully  paid  back  during  the  15-year  service  life  if  the  active  duty  units 
could  trade  off  about  8  percent  of  their  field  training  resources.  The 
reserve  units  would  have  to  trade  off  nearly  16  percent  of  their  training 
resources  to  break  even  within  the  same  period.  These  trade-offs  were 
estimated  without  frequency  data  from  the  reserves. 

Alternately,  the  preliminary  analysis  indicated  that  by  increasing  the  active 
duty  trade-offs  to  nearly  14  percent,  the  active  component  could  pay  for  its 
share  of  the  system  cost  and,  after  the  15-year  system  life,  accumulate  an 

*The  CCIT  development  contract  value  is  slightly  over  >409  million.  Program  officials  estimate  that  the 
16-year  life-cycle  cost  will  be  about  $1.3  billion,  much  of  which  is  operational  sustainment  costs. 


Uncertainty  About 
CCTT’s  Cost 
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estimated  savings  of  $771  million — an  amount  sufficient  to  cover  the 
$410-million  cost  to  produce  the  units  to  train  the  reserves.  The  study 
further  indicated  that  any  increase  in  cctt  life-cycle  costs  would  require 
further  trade-offs  in  traditional  field  training  resources  to  pay  back  the 
costs.  The  study  cautioned  that  the  extent  of  any  payback  would  have  to 
be  validated  based  on  the  results  of  user  testing. 

Uncertainty  over  the  Army’s  initial  cost  analysis  was  affirmed  by  the  use  of 
substantially  different  dollar  amounts  in  an  internal  affordability  briefing 
prepared  for  the  Army  Acquisition  Executive  in  July  1991 — 1  month  after 
the  milestone  I/EI  review.  Information  provided  in  that  briefing  showed 
the  e-cycle  cost  estimate  was  nearly  60  percent  higher  than  the  one 
»  i  in  the  preliminary  analysis.  Army  officials  attributed  part  of  the 
hi&i  er  costs  to  uncertain  estimates  of  sustainment  costs. 

While  the  reliability  of  cost  data  in  the  preliminary  analysis  is 
questionable,  documentation  associated  with  that  analysis  indicates  the 
Army’s  intent  to  assess  the  feasibility  of  offsetting  costs  of  the  cctt 
through  reduced  field  training  funds  in  the  future.  Further,  the  Army 
Acquisition  Executive  in  his  milestone  I/II  approval  memorandum  stated 
that  the  Army  was  committed  to  identifying  specific  field  training  costs 
reductions  that  are  made  possible  by  the  introduction  of  the  cctt  system. 
The  Army  Acquisition  Executive  also  stated  that,  where  practicable,  the 
savings  resulting  from  such  reductions  will  be  used  to  offset  the  costs  of 
developing  and  fielding  a  catt  program  and  to  expand  the  company  level 
capability  of  cctt  to  the  battalion  level. 

Program  officials  cautioned,  however,  that  the  Army  has  not  decided  how 
much  of  its  field  training  resources  it  is  willing  to  give  up  to  pay  for  the 
cctt  and  remained  firm  that  any  trade-ofis  should  begin  only  after  the 
system  is  in  the  field.  (The  system  is  currently  scheduled  to  be  fielded  in 
August  1998.)  This  uncertainty  reinforces  the  importance  of  operational 
testing  as  a  tool  for  providing  important  insights  into  cctt’s  cost  and 
training  effectiveness  before  the  milestone  in  review. 

The  preliminary  cost  analysis  indicated  that  the  active  component  costs 
could  be  repaid  over  the  ccrr’s  life  cycle,  but,  according  to  the  Army’s  cost 
analyst,  the  estimated  costs  of  preplanned  product  improvements  were  not 
included  in  the  analysis.  These  preplanned  improvements  are  the  desired 
features  that  program  managers  perceived  as  too  expensive  to  include  in 
the  basic  system  or  those  that  could  not  be  technologically  produced  at 
the  time.  They  include  simulation  for  such  features  as  electronic  warfare; 
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nuclear,  biological,  and  chemical  effects;  dynamic  terrain,*3  long-haul 
networks;  and  battalion  sets.  They  also  include  additional  modules  to 
simulate  the  armored  combat  earthmover  and  the  air  defense  antitank 
system. 

Although  the  documents  we  reviewed  identified  these  improvements,  their 
costs  were  not  estimated.  One  Army  official  involved  with  the  ccrr 
acquisition  said  these  preplanned  improvements  should  be  discussed 
during  the  milestone  m  review  to  fully  disclose  the  potential  system  costs; 
while  another  official  said  they  need  not  be  discussed  them  This  second 
official  said  each  improvement  should  be  reviewed  separately  as  it  is 
developed. 


Testing  Is  Critical  for 
Assessing  Cost  and 
Training  Effectiveness 


Although  the  ccrr  acquisition  plan  requires  operational  testing  to  be 
complete  prior  to  the  milestone  in  full-rate  production  decision,  some 
questions  exist  about  whether  testing  will  be  completed  as  planned.  An 
October  16, 1991,  CCTT  Test  and  Evaluation  Master  Plan  outlines  two  types 
of  required  tests:  technical  and  user.  The  technical  tests  are  needed  to 
determine  whether  ccrr’s  design  is  adequate  for  the  required  mission  and 
whether  it  meets  system  requirements.  User  tests  are  needed  to  evaluate 
the  effectiveness  of  CCTT  as  a  training  tool  and  determine  its  effect  on 
mission  performance. 


Technical  Tests  Technical  tests  are  scheduled  to  be  performed  over  a  period  of  about 

14  months  (beginning  in  July  1996)  by  both  the  contractor  and  the 
government  Testing  will  ensure  that  system  requirements  are  met  and  will 
include,  but  are  not  limited  to,  the  following  areas: 

•  cctt’s  capability  to  replicate  a  battlefield  for  conducting  combined  arms 
missions,  including  the  representation  of  different  types  of  simulators, 
semi-automated  friendly  and  opposing  forces,  command  and  control  work 
stations,  various  types  of  terrain,  and  acceptable  graphics; 

•  the  degree  of  fidelity  or  realism  of  soldier-machine  interfaces  necessary  to 
simulate  actual  combat  conditions; 

•  the  extent  to  which  safety  and  health  hazards  associated  with  operating  or 
maintaining  the  system  have  been  controlled  such  that  any  action  that 


’Dynamic  terrain  involves  the  simulation  of  changes  in  the  ground  or  terrain  as  ditches  are  dug  to 
provide  cover  for  tanks,  or  craters  are  created  from  the  effects  of  artilleiy  fire-  Dynamic  terrain 
remains  an  area  of  great  technical  challenge  that  has  the  potential  to  add  significantly  to  training 
capabilities. 
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might  cause  ir\jury  with  the  real  equipment  will  be  simulated  or  identified 
as  a  possible  safety  hazard;  and 

•  cctt’s  compliance  with  the  established  requirements  for  reliability, 
availability,  and  maintainability  such  that  system  failures  are  kept  to  a 
minimum  and  repairs  can  be  accomplished  within  an  acceptable  time 
frame. 

The  technical  tests  will  also  include  an  independent  verification  and 
validation  of  the  software  by  the  Army’s  Communications-Electronics 
Command’s  Center  for  Software  Engineering  to  ensure  the  clarity, 
completeness,  consistency,  and  traceability  of  the  software  to  the  system 
requirements.  The  Project  Manager  for  catt  will  be  responsible  for  seeing 
that  the  verification  and  validation  is  performed  concurrently  with  the 
software  development. 

According  to  Army  officials,  performing  the  verification  and  validation 
concurrent  with  the  software  development  should  help  identify  potential 
technical  problems  with  cctt  and  outline  solutions  before  the  system  is 
completed.  This  approach  is  consistent  with  a  recommendation  by  the 
Army’s  Test  and  Evaluation  Command  in  its  1991  assessment  of  cctt  risks. 


User  Tests  User  tests  are  currently  scheduled  to  be  performed  at  Fort  Hood,  Texas, 

beginning  in  December  1996.  This  testing  is  intended  to  evaluate  training 
effectiveness  through  a  series  of  company-sized  field  experiments  for  tank 
and  mechanized  infantry  units.4  Units  will  undergo  pre-training  evaluations, 
training,  and  post-training  evaluations  that  include  similar  situational 
exercises.  Between  evaluations  each  test  group  will  undergo  similar 
training  treatments  designed  to  evaluate  the  contribution  of  cctt  training 
to  combat  effectiveness  and  survivability.  One  group  will  receive  no 
training  and  will  serve  as  a  control. 

The  training  groups  and  mix  of  training,  as  summarized  in  table  4.2,  will 
include  various  mixes  of  traditional  field  training  and  cctt  simulation 
training  as  outlined  in  the  October  1991  Test  and  Evaluation  Master  Plan.6 


^Training  effectiveness  is  a  quantitative  measure  of  how  much  one’s  exposure  to  a  certain  training 
device  will  improve  or  hinder  performance.  This  measure  contrasts  with  training  transfer,  which 
observes  whether  training  positively  or  negatively  affects  how  one  performs  tasks,  but  does  not 
measure  the  degree  of  change  quantitatively. 

‘Army  officials  tokl  us  in  December  1992  that  they  are  examining  ways  of  streamlining  various  aspects 
of  the  planned  user  tests  to  control  costs  and  expedite  system  fielding.  However,  the  extent  of  those 
changes  and  their  impact  on  assessments  of  training  effectiveness  are  unclear  for  now. 
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A  detailed  plan  linking  the  functions  to  be  tested  with  the  test  scenarios  is 
scheduled  to  be  completed  and  approved  by  October  1995. 


Table  Training  Mix  for  CCTT 

Operational  Tests  Training  group  Training  mix 

1  Traditional  field  training 

2  2  days  CCTT ;  8  days  traditional  field 

_ training _ 

3  4  days  CCTT;  6  days  traditional  field 

_ training _ 

4  6  days  CCTT;  4  day.  ..aditional  field 

training 

5  _ No  training  (control  group) _ 


Our  review  of  the  master  test  plan  indicated  that  the  user  testing  section  is 
more  comprehensive  than  prior  simnet  evaluation  plans.  The  proposals  for 
simnet  field  tests  and  reports  we  reviewed  indicated  that  those  tests  were 
generally  subjective;  control  groups  were  not  used;  and  the  tests,  when 
conducted,  were  conducted  within  short  time  frames.  In  one  instance,  the 
simnet  tests  were  not  conducted  as  planned  because  of  high  costs  and  the 
unavailability  of  a  test  site.  Although  simnet  was  assessed  as  providing  a 
positive  training  benefit,  its  training  effectiveness  was  never  quantified. 
That  limitation  inhibited  the  potential  for  (1)  measuring  the  extent  of 
training  effectiveness  of  this  type  of  simulation  relative  to  traditional  field 
training  and  (2)  defining  the  contribution  of  each  to  the  most  appropriate 
mix  issue.  By  contrast,  cctt  tests  are  designed  to  measure  training 
effectiveness  and  determine  the  value  of  this  type  of  simulation  compared 
to  traditional  field  training. 

The  1991  cctt  Test  and  Evaluation  Master  Plan  estimated  that  about 
$15  million  will  be  needed  to  complete  the  technical  and  user  tests. 
Estimates  provided  by  Army  otticials  m  December  1992  indicated  the  costs 
should  be  less  than  $19  million. 

According  to  dod,  the  cctt  training  effectiveness  evaluation,  as  currently 
envisioned,  will  be  the  most  comprehensive  and  expensive  ever  devised  in 
dod.  However,  none  of  the  training  mixes  includes  a  ccrr-only  scenario, 
contrasted  with  a  field  training-only  scenario,  dod  believes  that  it  is  not 
needed  because  cctt  is  not  designed  as  a  replacement  for  field  training. 
Nevertheless,  such  a  comparison  might  give  even  greater  insight  into  the 
training  benefit  of  simulations  compared  with  more  traditional  forms  of 
training. 
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Production  of  Some 
CCTT  Units  Before 
Testing  and 
Production  Decision 


Because  of  pressures  within  the  Army  to  have  simulation  training 
capabilities  greater  than  those  available  through  simnet,  the  Army  decided 
to  field  in  Germany  and  the  United  States  68  limited-capability  “Quick 
Start”  cctt  units  in  advance  of  system  testing  and  a  production  decision.6 
Thus,  these  units  will  be  fielded  more  than  2  years  earlier  than  full 
capability  units  and  about  7  months  before  the  planned  initiation  of 
operational  testing. 

The  acting  Program  Manager  and  the  Training  and  Doctrine  Command’s 
System  Manager  said  they  believe  Quick  Start  was  a  good  way  to  get 
improved  training  capabilities  into  the  field  quickly.  They  indicated  that 
some  testing  of  Quick  Start  units  would  be  done  to  ensure  compliance 
with  technical  specifications.  Although  the  original  testing  plan  did  not 
include  assessments  of  training  effectiveness  for  the  Quick  Start  units,  we 
were  told  that  the  Army  now  plans  to  incorporate  such  tests. 


Caution  in 
Accelerated 
Development  and 
Fielding 


Differences  of  opinion  exist  within  the  Army  concerning  the  prospects  for 
further  accelerating  ccrr  development  and  production.  Some  Army 
officials  believe  that  the  acquisition  could  and  should  be  accelerated; 
others  point  to  significant,  and  time- consuming  development  activities  yet 
required,  such  as  producing  over  600,000  lines  of  computer  programming 
code.  A  program  official  told  us  that  one  way  to  accelerate  the  program 
would  be  to  reduce  the  time  allocated  for  testing.  In  December  1992,  Army 
officials  told  us  that  they  were  discussing  ways  to  accomplish  earlier  and 
more  constrained  testing  to  help  expedite  system  fielding. 

We  have  reported  extensively  on  the  importance  of  conducting  operational 
tests  and  evaluations  prior  to  beginning  full-rate  production.7  Failure  to  do 
so  can  result  in  higher  program  costs,  future  scheduling  delays  due  to  the 
need  to  make  system  modifications,  and  performance  problems. 


•The  Quick  Start  units  will  be  degraded  versions  of  the  CCTT,  relying  on  SIMNETs  communications 
capabilities  and  data  base.  Quick  Start  units  will  be  able  to  operate  in  a  stand-alone  mode  or  with 
SIMNET  simulators  to  incorporate  SIMNETs  semi-automated  forces.  The  full  production  CCTT  units 
will  have  significant  upgrades  of  all  of  these  features  requiring  extensive  software  development  Quick 
Start,  as  well  as  final  production  CCTT  units,  will  provide  tank  commanders  with  a  capability  lacking 
and  much  criticized  for  its  absence  in  SIMNET — a  simulated  popped  hatch  providing  the  commander  a 
360-degree  view  of  the  battlefield  as  though  he  were  able  to  look  at  the  battlefield  from  an  open  hatch 
on  the  top  of  the  tank.  Quick  Start  will  include  only  the  M1A1  tank  and  Bradley  Fighting  Vehicle;  the 
full-production  CCTT  units  will  simulate  additional  vehicles. 

TArmy's  M109  Howitzer  Required  Testing  Should  Be  Completed  Before  Full- Rate  Production 
(CA0/Ns1aT)-92-44,  Jan.  23,1 992)  and  Weapons  Testing:  TOP  Needs  to  Plan  and  Conduct  More  Timely 
Operational  Tests  and  Evaluations  (GaO-NSIaD-SO-107,  May  17, 1990).  This  report  also  lists  additional 
GAO  reports  on  this  suEJect 
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The  Army  is  working  toward  developing  a  strategy  to  integrate  ccrr  and 
other  catt  systems  with  traditional  field  training.  However,  the  lack  of 
data  on  system  cost  and  quantitative  assessments  of  how  much  each 
system  can  be  expected  to  contribute  to  overall  training  precludes  a 
definitive  answer  at  this  time  on  cctt’s  expected  cost-effectiveness  and  on 
the  most  cost-effective  mix  of  catt  systems  that  will  be  needed.  This 
uncertainty  makes  complete  performance  testing  and  cost  and  training 
effectiveness  analysis  even  more  important  Pressures  to  further 
accelerate  ccrr  development  and  production  could  prompt  the  Army  to 
shortcut  required  testing  and  analyses  before  making  full  production 
decisions.  However,  shortcutting  these  critical  reviews  could  limit  (1)  full 
disclosure  of  future  program  costs  for  the  ccrr,  including  possible  costs  of 
preplanned  product  improvements;  (2)  the  Army’s  ability  to  more 
definitively  assess  the  effectiveness  of  this  system  and  its  contribution  to 
the  right  mix  issue;  and  (3)  insight  into  potential  trade-offs  between 
simulations  and  resources  required  for  traditional  field  training. 


Recommendation 


We  recommend  the  Secretary  of  the  Army  ensure  that  all  prerequisite 
testing,  cost  analyses,  and  training  effectiveness  assessments  are 
completed  and  fully  considered  before  decisions  are  made  about  the 
full-rate  production  of  the  ccrr. 


Agency  Comments 


dod  concurred  with  our  recommendation.  It  noted  that  of  the  two  types  of 
required  testing  (technical  and  operational),  technical  testing  is  straight 
forward  and  can  be  executed  using  established  techniques,  dod  noted, 
however,  that  operational  testing  includes  an  unprecedented  critical  issue 
on  training  transfer.  Methodology  must  be  developed  to  determine  training 
transfer  in  a  combined  arms  environment  because  of  the  myriad  of 
variables  involved  in  combined  arms  operations. 
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Note:  A  fourth  milestone  may  be  necessary  to  determine  whether  significant  upgrades  are 
needed  for  the  weapon  in  production 
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Note:  GAO  comments 
supplementing  those  in  the 
report  text  appear  at  the 
end  of  this  appendix. 


OFFICE  OF  THE  DIRECTOR  OF 
DEFENSE  RESEARCH  AND  ENGINEERING 

WASHINGTON.  DC  20301 


?6 


Mr.  Frank  C.  Conahan 
Assistant  Comptroller  General 
National  Security  and  International 
Affairs  Division 
U.S.  General  Accounting  Office 
Washington,  D.C.  20548 


Dear  Mr.  Conahan: 


This  is  the  Department  of  Defense  (DoD)  response  to  the 
General  Accounting  office  draft  report  'DoD  AND  army  SIMULATION 
TRAINING:  Management  Framework  Improved,  But  Challenges  Remain', 

dated  February  4,  1993  (GAO  Code  391173/OSD  Case  9319).  The  DoD 
concurs  with  the  majority  of  the  findings  and  recommendations. 

The  DoD  only  partially  concurs  with  the  findings  on  replacing 
simulation  networking  with  the  Close  Combat  Tactical  Trainer,  the 
overlapping  roles  between  the  Joint  Staff  Offices,  and  the 
critical  testing  for  assessing  cost  and  training  effectiveness. 

The  DoD  also  only  partially  concurs  with  the  recommendation  that 
the  Secretary  of  Defense  eliminate  the  duplication  and  simulation 
inefficiencies  within  the  two  Joint  Staff  directorates  involved  in 
simulations  and  clearly  delineate  each  directorate's  role  and 
responsibilities  in  this  area. 


The  Joint  Staff  manages  modeling  and  simulation  activities  as 
an  integral  part  of  its  Information  Resources  Management  program. 
Specific  modeling  and  simulation  responsibility /proponency  are 
assigned  in  accordance  with  each  directorate's  function.  The 
Deputy  Director  for  Technical  Operations  is  responsible  for 
coordination  of  oversight,  planning,  and  management  of  modeling 
and  simulation  technology  for  the  Joint  Staff.  Studies  are 
currently  underway  to  clarify  the  use  and  application  of  modeling 
and  simulation.  Results  of  those  studies  are  expected  by  June 
1993. 


The  detailed  DoD  comments  on  the  recommendations  and  on  each 
finding  are  provided  in  the  enclosure.  The  DoD  appreciates  the 
opportunity  to  comment  on  the  draft  report. 


Enclosure 


Sincerely, 

CUu 

Jofyj  M.  BachHosky 
Deputy  Director, 
Defense  Research  and 
Engineering 
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Now  on  pp.  8-9. 


See  comment  1 


GENERAL  ACCOUNTING  OFFICE  DRAFT  REPORT 
DATED  FEBRUARY  4.  1993 

(GAO  CODE  391173)  OSD  CASB  9319 

"DOD  AND  ARMY  SIMULATION  TRAINING: 

MANAGEMENT  FRAMEWORK  IMPROVED,  BUT  CHALLENGES  REMAIN* 

DEPARTMENT  OF  DEFENSE  COMMBNTS 

***** 

FINDINGS 


•  FIND IMS — A:  Training  Devlcee  and  Slmulationa .  The 

GAO  reported  that  the  Military  Services  traditionally  have  used 
hundreds  of  training  aids  and  devices  to  model  or  simulate 
various  aspects  of  combat,  weapon  systems,  and  terrain  in 
support  of  training  activities.  The  GAO  found  that  computer 
simulations  are  growing  in  importance  as  training  devices, 
because  they  are  able  to  add  realism  to  training  and  potentially 
reduce  training  costs . 

The  GAO  observed  that  the  DoD  categorizes  simulations  into  the 
following  three  classes  on  the  basis  of  their  technical  and 
physical  structure: 

-  Computer  Simulation  Models- -Models  provide  a  view  of  the 

battle  on  a  map,  which  is  displayed  on  a  video  monitor 
with  units  on  the  map  shown  as  icons  (unit  symbols) . 
War  games  fall  into  this  category. 

-  Weapon,  system  simulators --Simulators  replicate 

the  performance  characteristics  and  appearance 
of  a  weapon  system  to  help  teach  individual 
skills. 

-  Instrumented  exercises --Actual  troops,  weapon  systems, 

and  support  systems  interact  in  as  real  an  environment 
as  possible,  short  of  combat,  (pp.  11-13/GAO  Draft 
Report) 

DOD  RESPONSE:  Concur.  The  definition  used  to  describe 
weapons  system  simulators  should  also  describe  crew  modules 
being  used  as  crew  trainers  (simulators)  and  as  platoon/company 
trainers  (simulations) . 
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elements  could  operate  together  on  a  regular  basis.  The  GAO 
observed  the  $200-million  effort  led  to  the  creation  of  a 
training  system  that  came  to  be  called  Simulation  Networking. 

The  GAO  noted  that,  in  Simulation  Networking,  numerous 
computer-driven  simulators  (configured  around  a  mockup  of  the 
interior  of  an  armored  vehicle)  are  electronically  linked  to 
form  a  common  combat  arena  or  battlefield. 

The  GAO  explained  that  Simulation  Networking  also  incorporates 
some  battlefield  participants  that  are  generated  by  the 
computer.  The  GAO  observed  that  the  participants  are  called 
semi -automated  forces  because  a  trainer  initiates  a  combat  task 
from  a  computer  workstation,  but  the  task  is  performed 
automatically  according  to  specified  war-fighting  doctrine  and 
tactics  programmed  into  the  system.  The  GAO  noted  that 
Simulation  Networking  also  has  a  built-in  data  collection  and 
analysis  system,  which  records  a  complete  history  of  a  simulated 
battle--including  (1)  orders  to  take  actions,  (2)  the  resulting 
actions,  and  (3)  the  impact  of  the  actions. 

The  GAO  reported  that  Simulation  Networking  was  first 
demonstrated  in  May  1986,  at  the  Army's  Armor  Center  in  Fort 
Knox,  Kentucky.  The  GAO  noted  that,  ultimately,  a  total  of  236 
simulators  were  acquired  and  placed  in  eleven  sites  in  the 
United  States  and  Germany,  where  they  have  remained  in  continual 
use  as  a  training  device.  The  GAO  pointed  out  that  Army  forces 
used  Simulation  Networking  to  prepare  for  Operation  Desert 
Storm. 

The  GAO  indicated  that  the  success  of  the  Simulation  Networking 
program  has  led  to  the  formalization  of  the  networking  protocols 
as  an  industry  standard  known  as  Distributive  Interactive 
Simulation  standards.  The  GAO  noted  that  the  Distributive 
Interactive  Simulation  concept  has  been  expanded  into  several 
other  Service-specific  and  DoD-wide  initiatives.  The  GAO  noted, 
for  example,  that  Air  Force  Special  Operations  are  developing  a 
distributive  interactive  simulation-based  air  crew  training 
system  that  will  allow  multiple  aircraft  to  interact  in  a 
simulated  battle. 

The  GAO  reported  that  several  studies  undertaken  by  the  Army 
Training  and  Doctrine  Command  to  assess  the  Simulation 
Networking  training  benefit  found  the  following  limitations: 

-  only  two  major  combat  vehicles  were  represented; 
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-  vehicle  movements  were  not  impeded  by  trees,  rivers,  or 
buildings,  and  vehicle  drivers  received  no  meaningful  speed 
cues  from  the  terrain; 

-  fields  of  view  from  the  video  monitors  were  smaller  than 
in  actual  vehicles,  and  the  gunnery  systems  were  unable  to 
identify  targets  at  realistic  distances;  and 

-  the  Commanders'  view  of  the  battlefield  was  constrained. 

In  addition,  the  GAO  reported  that  an  independent  assessment  of 
the  Simulation  Networking  software  by  a  systems  engineering 
company  also  found  many  technical  errors  in  computer  code 
Now  on  pp. 10-13.  written  to  depict  system  performance,  (pp.  15-20/GAO  Draft 

Report) 

POD  RESPONSE :  Concur.  The  reference  to  the  Air  Force  use  of 
the  Distributive  Interactive  Simulation  standard  for  Special 
Operations  is,  however,  over  stated.  The  Department  has 
directed  the  Air  Force  and  the  Special  Operations  Command  to  use 
the  Distributive  Interactive  Simulation  but,  because  the 
See  Comment  1.  Distributive  Interactive  Simulation  standard  is  relatively 

immature,  and  lacks  a  standard  set  of  dead  reckoning  algorithms 
for  fast  movers,  it  is  likely  this  will  be  done  as  a 
modification  at  a  later  date. 

As  a  result  of  the  Army  experience  in  Simulator  Networking  and 
the  subsequent  leadership  in  the  Distributive  Interactive 
Simulation  standard,  the  Army  has  participated  in  international 
simulation  forums.  That  participation  has  led  to  the  writing  of 
Standard  Agreements  to  establish  the  Distributive  Interactive 
Simulation  standard  as  an  international  military  standard. 

In  the  GAO  discussion  on  the  limitations  of  Simulator 
Networking,  references  were  made  to  the  number  of  vehicles  and 
vehicle  movement.  For  the  number  of  vehicles,  a  more  accurate 
statement  would  be  "only  two  major  ground  combat  vehicles  were 
represented.'  Attack  Helicopters  and  A-10  Thunderbolt  aircraft 
were  also  networked  during  the  proof  of  principle  tests.  When 
impediments  were  discussed,  the  GAO  states  buildings  and  rivers 
did  not  halt  combat  vehicles.  That  is  not  a  correct  statement. 


•  FINPma  P:  Replacing _ Simulation _ Networking _ with _ cioae 

Pnmhat;  Tactical  Trainer.  The  GAO  observed  that,  even  before 
assuming  full  responsibility  for  Simulation  Networking  in  1990 
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from  the  Defense  Advanced  Research  Projects  Agency,  the  Army  was 
already  addressing  Simulation  Networking  deficiencies.  The  GAO 
noted  that  the  Army  wanted  to  develop  a  family  of  advanced 
simulators  to  train  all  of  its  forces  under  an  umbrella  concept 
known  as  the  Combined  Arms  Tactical  Training  system.  The  GAO 
further  noted  that  the  first  device  is  the  Close  Combat  Tactical 
Trainer  for  armor  and  mechanized  infantry  forces.  The  GAO 
explained  that  the  Close  Combat  Tactical  Trainer  is  expected  to 
be  followed,  in  sequence,  by  (1)  an  Aviation  Combined  Arms 
Tactical  Trainer,  (2)  an  Air  Defense  Combined  Arms  Tactical 
Trainer,  (3)  an  Engineering  Combined  Arms  Tactical  Trainer,  and 
(4)  the  Closed  Loop  Artillery  Simulation.  The  GAO  further 
explained  that  the  Army  is  also  considering  other  potential 
systems  for  intelligence  and  electronic  warfare.  (pp.  20-21/ 
GAO  Draft  Report) 

POD  RESPONSE;  Partially  concur.  The  Army  has  no  established 
sequence  for  follow-on  trainers  as  part  of  the  Combined  Arms 
Tactical  Trainer  family. 


Nowon  pp.  14-15. 


•  finding  E.-  Concerns _ Beaaxdlng _ Simulation  Management . 

The  GAO  reported  that,  since  1976,  reports  by  various 
DoD  and  Military  Service  study  panels  and  independent  audit 
groups  have  reflected  concerns  about  the  management  of  modeling 
and  simulation  activities.  The  GAO  noted  that  a  1988  study  by 
the  Defense  Science  Board  Task  Force  identified  many  problems 
with  the  management  of  modeling  and  simulation  activities.  The 
GAO  mentioned,  for  example,  that  the  study  revealed  that  Service 
and  individual  joint  commanders  did  not  coordinate  with  each 
other  when  building  or  modifying  automated  training 
simulations- -and,  therefore,  built  duplicate  models.  The  GAO 
also  noted  that  the  House  Armed  Services  Committee  expressed 
concerns  about  the  adequacy  of  DoD-wide  coordination  involving 
computer  simulation  technology  that  in  its  August  1990 
Authorization  Report  for  Fiscal  Year  1991  Defense 
Authorizations.  The  GAO  added  that  the  Committee  recommended  the 
establishment  of  a  central  office  to  coordinate  the  technology 
throughout  the  DoD  and  raised  questions  about  the  adequacy  of 
the  Close  Combat  Tactical  Trainer  program  management,  (pp. 
21-23/GAO  Draft  Report) 


Concur. 
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•  EIMCJMS — £:  Efforts  to  Strengthen  Coordination  and 
Management  of  simulation  Activities.  The  GAO  reported  that, 
in  June  1991,  in  response  to  the  growing  concerns  about  the 
inadequacy  of  DoD  coordination  and  the  lack  of  guidance  dealing 
with  simulation  activities,  the  Deputy  Secretary  of  Defense 
established  the  Defense  Modeling  Simulation  Office.  The  GAO 
observed  that  the  Defense  Modeling  Simulation  Office  operates 
under  the  guidance  of  a  high-level  executive  council,  chaired  by 
the  Director  of  Defense  Research  and  Engineering.  The  GAO  noted 
that  in  FY  1993,  the  Defense  Modeling  Simulation  Office  expects 
to  issue  a  DoD  directive  outlining  requirements  for  a  DoD-wide 
approach  for  developing  and  using  models  and  simulations.  The 
GAO  also  noted  that  a  draft  of  the  directive  requires  the 
Military  Services  and  the  DoD  organizations  to  implement 
management  systems  to  oversee  their  modeling  and  simulation 
activities,  and  to  facilitate  interservice  coordination.  The  GAO 
determined  that  the  Defense  Modeling  Simulation  Office  is 
seeking  to  foster  the  development  of  Defense-wide  standards, 
data  bases,  and  communications  capabilities  in  the  simulation 
area,  and  the  Defense  Modeling  Simulation  Office  is  pursuing  the 
goals  through  functional  and  technical  working  groups  and  by 
funding  various  simulation  projects  submitted  by  the  Services, 
the  Joint  Staff,  and  Defense  Agencies,  (pp.  26-30/GAO  Draft 
Report) 

POP  RESPONSE :  Concur . 


•  riMPiMQ — a=  Heed  for  Lana  Range  Man ter  Plaa  and 

Controls  Over  Simulation  Development .  The  GAO  reported  that 
in  the  May  1992  Defense  Modeling  And  Simulation  Initiative 
management  plan,  the  Defense  Modeling  and  Simulation  Office 
outlined  its  vision,  goals,  objectives,  investment  strategy,  and 
future  activities.  In  this  plan,  the  GAO  indicated,  the  Defense 
Modeling  and  Simulation  Office  recognizes  that  the  DoD  does  not 
have  a  unified  statement  of  modeling  and  simulation  needs  and 
deficiencies  across  the  DoD.  To  help  identify  needs  and 
deficiencies  in  the  simulation  area,  the  GAO  observed,  the 
Defense  Modeling  and  Simulation  Office  is  conducting  workshops 
with  representatives  of  functional  communities,  such  as  training 
and  test  and  evaluation,  from  each  of  the  Services,  the 
Commander  in  Chief,  Defense  Agencies,  and  Federally  Funded 
Research  and  Development  Centers.  The  GAO  mentioned  that,  by 
March  1993,  the  Defense  Modeling  and  Simulation  Office  expects 
to  summarize  and  publish  the  results  of  the  workshops. 
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Now  on  pp.  20-21. 


The  GAO  also  reported  that  the  Defense  Modeling  and  Simulation 
Office  and  other  simulation  experts  recognize  the  need  to 
improve  controls  over  the  development  and  use  of  models  and 
simulations.  The  GAO  assessed  that  two  areas  of  particular 
concern  to  the  Defense  Modeling  and  Simulation  Office  are 
configuration  management  and  the  verification,  validation,  and 
accreditation  of  models.  The  GAO  reported  that,  in  beginning  to 
address  these  issues,  the  Defense  Modeling  Simulation  Office  has 
included  in  a  draft  DoD  directive  on  models  and  simulations  a 
requirement  that  the  Military  Services  and  other  Defense 
organizations  assign  responsibility  for  configuration  control  to 
a  specific  organization.  The  GAO  also  noted  that  the  Defense 
Modeling  and  Simulation  Office  has  included  in  its  draft 
directive  a  requirement  for  the  Services  to  establish  a 
verification,  validation,  and  accreditation  program,  (pp.  30-32/ 
GAO  Draft  Report) 

POP  RBSFOMSB:  Concur.  The  workshop  results  were  published  as 
a  Defense  Technical  Information  Center  report  in  March  1993 . 

The  draft  Department  of  Defense  Directive  is  in  the  final  stages 
before  being  issued.  Approval  by  the  Secretary  of  Defense  is 
anticipated  by  May  1993. 


•  riMPIHfl._a»  Defense  Modeling  and  Simulation  Office 
Staffing  and  funding.  The  GAO  found  that  the  Defense  Modeling 
and  Simulation  Office  lacks  permanent  personnel  billets  and 
operates  with  a  temporary  staff  that  has  fluctuated  in  size  from 
approximately  six  to  eight  Government  personnel  on  loan  from  the 
Services.  The  GAO  determined  chat  the  staff  is  augmented  by  up 
to  eight  staff  years  of  contractor  support.  The  GAO  reported 
that  Defense  Modeling  and  Simulation  Office  officials  have 
sought,  but  not  yet  obtained,  some  permanent  staffing  authority. 
Although  the  Congress  strongly  supported  the  Defense  Modeling 
and  Simulation  Office  by  creating  and  initially  funding  it,  the 
GAO  speculated  that  internal  DoD  budget  reviews  have  created 
uncertainty  about  future  funding  prospects,  (pp.  32-34/GAO  Draft 
Report ) 

POP  B8SEQM3I1  Concur.  The  provision  of  permanent  personnel 
billets  is  being  pursued.  A  military  (06)  billet  has  already 
been  provided  and  will  be  used  for  the  DKSO  Director.  Funding 
has  been  stabilized  with  the  inclusion  of  the  Defense  Modeling 
end  Simulation  Office  in  the  Program  Objective  Memorandum. 
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off lean .  The  GAO  observed  that,  in  contrast  with  the  Defense 
Modeling  and  Simulation  Office,  which  is  filling  a  broader 
mandate  and  more  of  an  integrating  role  across  the  Military 
Services,  the  efforts  of  the  Joint  Staff  focus  on  joint 
operational  matters  involving  the  Commanders  in  Chief.  The  GAO 
concluded,  however,  that  two  separate  Joint  Staff  directorates 
have  overlapping  responsibilities  in  providing  simulation 
assistance  to  the  commanders  in  Chief  and  each  brings  a 
different  perspective  on  how  to  provide  such  support. 

The  GAO  found  that  J-7  oversees  operations  of  the  Joint  warfare 
Center — and,  in  March  1992,  J-7  established  a  Joint  Simulation 
and  Interoperability  Division  and  tasked  it  to  coordinate 
modeling  and  simulation  activities  involving  the  Commanders  in 
chief.  The  GAO  observed  that  the  Joint  Simulation  and 
Interoperability  Division  mission  is  to  promote  the  application 
of  models  and  simulations  in  joint  operational  planning  and 
execution,  education,  training,  exercises,  operations 
requirements,  joint  test  and  evaluation,  and  doctrine 
development  and  evaluation. 

The  GAO  furcher  found  that,  at  the  same  time,  J-S  is  tasked  with 
developing  joint  modeling  and  simulation  capabilities  as 
necessary  to  support  assigned  missions,  including  conducting 
joint  war  games  and  interagency  politico-military  seminars  and 
simulations.  The  GAO  determined  that  J-8  assigned  missions  were 
not  changed  with  the  creation  of  the  Joint  Simulation  and 
interoperability  Division.  The  GAO  observed  that,  according  to 
the  J-8  technical  director,  J-8  has  long  been  responsible  for 
providing  or  arranging  for  simulation  support  to  the  Commanders 
in  Chief  across  a  broad  spectrum  of  functions,  including 
training,  education,  operational  planning,  and  analysis.  The  GAO 
also  reported  that  J-8  is  the  focal  point  for  the  Joint  Staff 
Modern  Aids  to  Planning  Program- -through  which  Commanders  in 
Chief  obtained  their  own  simulation  capabilities  (hardware  and 
software) . 

The  GAO  noted  one  area  of  long  standing  concern  to  the  Congress 
has  been  the  lack  of  a  clear  assessment  of  Commander  in  Chief 
simulation  requirements.  The  GAO  found  that  both  of  the  Joint 
Staff  directorates  claim  some  responsibility  for  completing  that 
assessment,  and  both  directorates  have  been  surveying  the 
Commanders  in  Chief  to  determine  their  simulation  needs.  The  GAO 
also  observed  that,  at  the  same  time,  some  Commanders  in  Chief 
have  expanded  their  own  simulation  capabilities  from  other 
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Commanders  in  Chief  to  determine  their  simulation  needs.  The  GAO 
also  observed  that,  at  the  same  time,  some  Commanders  in  Chief 
have  expanded  their  own  simulation  capabilities  from  other 
simulation  centers  operated  by  the  Army  and  Air  Force.  The  GAO 
concluded,  therefore,  that  at  the  present  time,  simulation 
support  for  Commanders  in  Chief  activities  comes  from  multiple 
sources  within  and  external  to  the  Joint  Staff.  The  GAO  did  note 
that  J-7  is  currently  conducting  a  study  to  assess  whether  a 
consolidated  joint  training,  doctrine,  and  simulation  command  is 
needed.  In  addition,  the  GAO  found  two  other  efforts  are 
underway  that  might  eliminate  some  of  the  overlap: 

-  the  Vice  Director  of  the  Joint  Staff  plans  to  initiate, 
in  March  1993,  a  two-month  study  of  the  organization  and 
staffing  of  the  Joint  Staff;  and 

-  the  Chairman,  in  response  to  legislative  requirements,  is 
conducting  a  comprehensive  study  of  the  roles  and  missions 
of  the  Armed  Forces,  (pp.  34-39/GAO  Draft  Report) 

POD  RESPONSE:  Partially  concur.  One  of  the  ongoing  Joint 
Staff  studies,  expected  to  be  completed  in  June  1993,  is  being 
conducted  under  the  direction  of  the  Director,  Joint  Staff. 

Part  of  that  study  concerns  whether  a  consolidated  joint 
training,  doctrine,  and  simulation  'center'  is  needed.  The 
study  is  not  addressing  a  consolidated  joint  training,  doctrine, 
and  simulation  'command'. 

Joint  Administrative  Publication  1.1,  'Organization  of  and 
Functions  of  the  Joint  Staff,'  assigns  modeling  and  simulation 
responsibility /proponency  to  several  Joint  Staff  directorates, 
including  J-4  (Logistics),  J-6  (Command  and  Control),  J-7 
(Interoperability  and  Training),  and  J-8  (Force  Structure  and 
Assessment) .  Directorate  modeling  and  simulation  activity 
within  those  assigned  areas  is  appropriate  and  does  not 
constitute  duplication  or  overlap  of  effort. 

The  GAO  report  accurately  reflects  the  status  of  ongoing  Joint 
Staff  studies  that  will  clarify  the  use  and  application  of 
modeling  and  simulation  technology  in  support  of  the  Chairman  of 
the  Joint  Chiefs  of  staff,  the  Joint  Staff,  and  the  Unified  and 
Specified  Commanders.  These  efforts  will  help  maximize  the 
utilization  of  ever  scarcer  modeling  and  simulation  resources. 
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*  FINDING _ ii.  Army  Strengthens  Simulation  Management _ and 

Ensure  System  Interoperability .  The  GAO  observed  that,  in 
responding  to  concerns  about  the  management  of  the  Combat 
Tactical  Trainer  and  other  simulation  systems,  the  Army 
strengthened  its  management  structure  for  acquiring  simulations. 
The  GAO  noted  that  the  revised  structure  is  intended  to  elevate 
acquisition  decisions  to  a  higher  level  and  to  ensure  more 
direct  involvement  of  the  Army  Acquisition  Executive--a  role 
filled  by  the  Assistant  Secretary  of  the  Army  for  Research, 
Development,  and  Acquisition. 

The  GAO  reported  that,  in  November  1991,  the  Army  developed  a 
master  plan  to  outline  the  management  structure  for  acquiring 
the  Close  Combat  Tactical  Trainer  and  other  simulations  in  the 
Combined  Arms  Tactical  Training  system.  The  GAO  observed  that 
the  master  plan  stated  Combined  Arms  Tactical  Training  is  not  a 
single  training  system--but  is,  instead,  a  series  of  unique 
training  devices  for  each  of  the  Army's  combat  forces — for 
example  armor,  aviation,  air  defense,  artillery.  The  GAO  noted 
that  future  Combined  Arms  Tactical  Training  component 
acquisitions  are  expected  to  follow  the  same  acquisition  process 
as  the  Close  Combat  Tactical  Trainer.  The  GAO  also  noted  that, 
because  of  its  estimated  cost  in  excess  of  $1  billion  and  high 
visibility,  the  Close  Combat  Tactical  Trainer  was  designated  as 
a  category  II  major  defense  acquisition  program.  Such  a 
designation,  the  GAO  mentioned,  requires  the  Army  Acquisition 
Executive  to  approve  key  acquisition  decisions  following 
milestone  reviews  by  the  Army  Systems  Acquisition  Review 
Council.  (In  report  table  3.1,  the  GAO  delineated  Close  Combat 
Tactical  Trainer  milestones  as  of  December  1992.) 

Like  Close  Combat  Tactical  Trainer,  the  GAO  reported  each 
follow-on  Combined  Arms  Tactical  Training  component  is  expected 
to  be  acquired  as  a  major,  separate,  stand-alone  system  with 
funding  budgeted  as  a  separate  program  line  item.  The  GAO  also 
reported  that,  on  November  30,  1992,  the  Army  awarded  a 
contract- -valued  at  $409  million--to  International  Business 
Machines,  Inc.,  to  begin  full-scale  development  of  the  Close 
Combat  Tactical  Trainer.  The  GAO  observed  that,  in  August  1992, 
the  Army  created  a  new  major  subordinate  command- -known  as  the 
Simulation,  Training,  and  Instrumentation  Command--to  centralize 
oversight  of  technology  development  and  life  cycle  support  for 
training  devices--i.e. ,  instrumentation,  targets,  and  threat 
simulations,  as  well  as  other  Army  battlefield  simulations.  The 
GAO  noted  that  the  new  structure  provides  more  direct  lines  of 
communication  with  the  Army  Acquisition  Executive  in  major 
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Now  on  p.  28. 


decisions  affecting  Combined  Arms  Tactical  Training  components 
The  GAO  also  noted  that  the  Army  appointed  a  separate  Project 
Manager  for  Combined  Arms  Tactical  Training  to  manage  the 
development  and  fielding  of  the  Close  Combat  Tactical  Trainer 
and  other  Combined  Arms  Tactical  Training  systems,  (pp.  41-46/ 
GAO  Draft  Report) 

POD  RESPONSE :  Concur.  For  accuracy,  the  words  ‘full-scale 
development”  used  to  describe  the  acquisition  of  the  Close 
Combat  Tactical  Trainer  should  be  changed  to  "engineering  and 
manufacturing  design" . 


•  EXHBIHS — X:  Steps _ Taken _ to  Ensure _ Interoperability 

Among _ Combined _ Anna  Tactical  Training  Components. 

The  GAO  reported  that  the  Army  has  taken  steps  to  ensure 
interoperability  and  compatibility  among  Combined  Arms  Tactical 
Training  components  by  (1)  mandating  the  use  of  the  DoD  standard 
military  programming  language  and  (2)  requiring  an  "open  systems 
design"  where  interfaces  between  hardware  and  software  do  not 
depend  on  vendor-specific  or  proprietary  equipment.  The  GAO 
further  reported  that,  moreover,  the  Army  is  taking  a  leadership 
role  in  pursuing  the  development  of  standards  to  govern  future 
Army  and  DoD  simulations  and  ensure  their  interoperability.  The 
GAO  concluded  that  the  Army  has  included  requirements  in  the 
Combined  Arms  Tactical  Training  master  plan  and  the  Close  Combat 
Tactical  Trainer  specifications  to  ensure  interoperability  among 
simulator  systems,  (p.  46/GAO  Draft  Report) 

POP  RESPONSE:  Concur. 


*  FltiDlRQ — It-  Lack  of  Validated  Data  on  Mix  of 
Simulations  and  Field  Training.  The  GAO  also  reported  that 
the  Army  is  developing  a  combined  arms  training  strategy  to  help 
manage  its  training  resources  in  a  more  integrated  manner.  The 
GAO  observed  that  the  strategy--to  establish  a  definitive 
relationship  between  the  mix  of  field  and  simulation 
training--is  still  evolving,  and  it  is  not  clear  when  definitive 
data  will  be  available  to  provide  precise  answers  to  questions 
about  (1)  the  most  appropriate  mix  of  field  and  simulation 
training,  (2)  the  best  combination  of  Combined  Arms  Tactical 
Training  systems  the  Army  will  need,  or  (3)  the  extent  to  which 
they  will  be  linked  together  to  train  higher  echelons.  The  GAO 
also  observed  that  plans  for  the  Close  Combat  Tactical  Trainer 
and  other  Combined  Arms  Tactical  Training  systems  initially 
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focused  on  integrating  simulations  and  field  training  at  the 
battalic  level.  The  found,  however,  that  the  Army  now  plans 
to  use  tne  Close  Combat  Tactical  Trainer  at  the  platoon  and 
Company  levels  due,  in  part,  to  cost  considerations. 

The  GAO  noted  that  individual  training  strategies  are  being 
developed  by  the  Army  training  centers  and  schools  to  identify 
the  best  mix  of  training  resources  and  to  help  ensure  that  the 
training  effectiveness  of  a  training  device  is  determined  t  ‘ore 
critical  acquisition  decisions  are  made. 

In  report  table  4.1,  the  GAO  summarized  the  quantities  and  the 
delivery  schedule  for  the  estimated  546  total  Close  Combat 
Tactical  Trainer  units  needed  to  train  the  active  and  reserve 
armor  units  at  the  echelons  of  company  level  and  below.  The  GAO 
acknowledged  that  Army  officials  recognize  their  numbers  are 
tentative  and  must  be  validated  through  user  testing.  The  GAO 
identified  the  following  three  key  questions  that  should  be 
answered  from  user  testing: 

-  how  long  and  at  what  intervals  should  units  train  on  the 
Close  Combat  Tactical  Trainer; 

-  to  what  extent  should  the  Close  Combat  Tactical  Trainer 
be  used  as  a  prerequisite  to  certain  field  exercises;  and 

-  to  what  extent  should  the  Close  Combat  Tactical  Trainer 
be  used  to  reinforce  or  sustain  skills  previously  acquired. 

The  GAO  explained  that,  in  developing  its  acquisition  plan,  the 
Army  subjectively  decided  to  configure  20  percent  of  the  total 
Close  Combat  Tactical  Trainer  units  as  mobile  sites.  The  GAO 
further  explained  that  a  subjective  decision  was  necessary 
because  the  Army  did  not  have  experience  with  mobile  units  to 
provide  a  more  informed  basis  for  its  decision.  The  GAO 
acknowledged  that  the  Army  recognizes  the  most  appropriate  mix 
between  field  training  days  and  simulation  time  for  other 
Combined  Arms  Tactical  Training  systems  also  is  still  subject  to 
validation.  The  GAO  observed  that  the  ultimate  mix  of  Combined 
Arms  Tactical  Training  systems  is  unclear  at  this  time,  because 
the  Army  does  not  know  how  often  the  Close  Combat  Tactical 
Trainer  and  other  systems  will  be  used  independently  or 
networked  at  higher  echelons  and  because  it  has  yet  to  validate 
the  affordability  of  the  complete  Combined  Arms  Tactical 
Training  system,  (pp.  47-52/GAO  Draft  Report) 
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POD  RESPONSE :  Concur. 


•  FINDING  M;  Pncertalntv  About _ Clone _ Combat _ Xa.CtA.CAl 

Trainer's  coat.  The  GAO  reported  that  the  Army  System 
Acquisition  Review  Council  tentatively  approved  the  Close  Combat 
Tactical  Trainer  development  at  the  June  1991  combined  milestone 
X/II  review,  in  part,  on  the  basis  of  preliminary  cost  and 
training  effectiveness  data  that  Army  officials  recognized  as 
"soff  and  requiring  further  validation.  The  GAO  pointed  out 
that,  although  the  preliminary  cost  analysis  concluded  that  the 
Close  Combat  Tactical  Trainer  had  the  potential  to  be  a  cost- 
effective  addition  to  the  Army  training  program  and  could 
probably  pay  for  itself,  it  was  also  recommended  that  the 
savings  be  quantitatively  validated  based  on  the  result  of  user 
testing.  The  GAO  also  reported  the  October  1991  review  council 
approval  stated  that  potent- V  reductions  in  training  resources 
due  to  the  Close  Combat  Ta<  "  -  Trainer  must  be  validated  by 
test  results  and  not  based  on  artificial  numbers.  The  GAO 
observed  that,  cost  and  training  effectiveness  are  scheduled  to 
be  reassessed  before  the  Army  decides  whether  to  proceed  with 
the  full  production  of  the  Close  Combat  Tactical  Trainer  in 

1997. 

The  GAO  noted  that,  while  the  reliability  of  the  cost  data  in 
the  preliminary  analysis  is  questionable,  documentation 
associated  with  that  analysis  indicates  the  Army's  intent  to 
assess  the  feasibility  of  offsetting  costs  of  the  Close  Combat 
Tactical  Trainer  through  reduced  field  training  funds  in  the 
future.  Further,  the  GAO  noted,  the  Army  Acquisition  Executive, 
in  his  milestone  I/II  approval  memorandum,  stated  that  (1)  the 
Army  was  committed  to  the  identification  of  specific  field 
training  costc  reductions  that  are  made  possible  by  the 
introduction  of  the  Close  Combat  Tactical  Trainer  system,  and 
(2)  where  practicable,  the  savings  resulting  from  such 
reductions  will  be  used  to  offset  the  costs  of  developing  and 
fielding  a  Combined  Arms  Tactical  Training  program  and  to  expand 
the  company  level  capability  of  Close  Combat  Tactical  Trainer  to 
the  battalion  level.  The  GAO  noted  that  Program  officials 
cautioned,  however  that  the  Army  has  not  decided  how  much  of  its 
field  training  resources  it  is  willing  to  give  up  to  pay  for  the 
CLose  Combat  Tactical  Trainer  system- -and  remained  firm  that  any 
tradeoffs  should  begin  only  after  the  system,  is  in  the  field. 
The  GAO  reported  that  system  fielding  is  scheduled  for  August 

1998.  Although  the  preliminary  cost  analysis  indicated  the 
system  could  pay  for  itself,  the  GAO  found  that  the  estimated 
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costs  of  preplanned  product  improvements  were  not  included  in 
the  analysis.  The  GAO  indicated  the  preplanned  improvements 
include  simulation  for  such  features  as  (1)  electronic  warfare, 
(2)  nuclear,  biological  and  chemical  effects,  (3)  dynamic 
terrain,  (4)  long-haul  networks,  (5)  battalion  sets,  and  (6) 
additional  modules  to  simulate  the  armored  combat  earthmover  and 
the  air  defense  antitank  system. (pp.  53-56/  GAO  Draft  Report) 

POD  RESPONSE ;  Concur. 


♦  f  IHBIH.B — H:  I.Bfl&lag _ la  Critical  for  Assessing  Coat  and 

Training  Effectiveness.  The  GAO  reported  that  the  Close 
Combat  Tactical  Trainer  acquisition  plan  calls  for  formal 
operational  and  performance  testing  prior  to  the  milestone  III 
production  decision.  The  GAO  observed  that  technical  tests  by 
both  the  contractor  and  the  Government  are  scheduled  to  be 
performed  over  a  period  of  about  14  months  (beginning  in  June 
1995) .  The  GAO  noted  the  testing  will  ensure  that  system 
specifications  are  met.  The  GAO  further  noted  that  the 
technical  tests  will  also  include  an  independent  verification 
and  validation  of  the  software  by  the  Army  Test  and  Evaluation 
Command  to  ensure  the  clarity,  completeness,  consistency,  and 
traceability  of  the  software  to  the  system  requirements.  The 
GAO  explained  that  the  Project  Manager  for  Combined  Arms 
Tactical  Training  will  be  responsible  for  ensuring  that  the 
verification  and  validation  is  performed  properly  and 
concurrently  with  the  software  development,  starting  shortly 
after  the  contract  is  awarded  and  continuing  into  full 
production.  The  GAO  speculated  that  performing  the  independent 
verification  and  validation  concurrently  with  the  software 
development  should  help  identify  potential  technical  problems 
and  outline  solutions  before  the  system  is  completed. 

The  GAO  also  reported  that  user  tests,  also  called  operational 
tests,  are  currently  scheduled  to  be  performed  at  Fort  Hood, 
Texas,  beginning  in  December  1996.  The  GAO  noted  that  testing  is 
intended  to  evaluate  training  effectiveness  through  a  series  of 
company-sired  field  experiments  for  tank  and  mechanized  infantry 
units.  The  GAO  summarized  in  Table  4.2,  the  various  mixes  of 
traditional  field  training  and  Close  Combat  Tactical  Trainer 
simulation  training  as  outlined  in  the  October  1991  Test  and 
Evaluation  Master  Plan.  The  GAO  reported  that  a  detailed  plan 
linking  the  functions  to  be  tested  with  the  test  scenarios  is 
scheduled  to  be  completed  and  approved  by  October  1995. 
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See  comment  4. 


The  GAO  concluded,  however,  that  one  limitation  of  the  Close 
Combat  Tactical  Trainer  test  plan  is  that  none  of  the  training 
mixes  includes  a  Close  Combat  Tactical  Trainer-only  scenario, 
contrasted  with  a  field  training-only  scenario.  The  GAO  asserted 
that  such  a  comparison  might  give  even  greater  insight  into  the 
training  benefit  of  simulations  compared  with  the  more 
traditional  forms  of  training.  The  GAO  pointed  out  its  review  of 
the  master  test  plan  indicated  that  the  user  testing  section  is 
more  comprehensive  than  prior  Simulation  Networking.  The  GAO 
referenced  that  the  1991  Close  Combat  Tactical  Trainer  test  and 
evaluation  master  plan,  which  estimated  that  about  $15  million 
will  be  needed  to  complete  the  technical  and  user  tests,  but 
that  estimates  provided  by  Army  officials  in  December  1992 
indicated  the  costs  could  range  upwards  to  $19  million. (pp. 
56-61/GAO  Draft  Report) 

pod  RHSPONSB:  Partially  concur.  The  Army's  training 
effectiveness  evaluation  is  the  most  comprehensive  ever  devised 
in  the  DoD.  The  greatest  limitation  will  be  in  (1)  acquiring 
sufficiently  large  groups  to  ascertain  statistically  significant 
results  and  (2)  controlling  or  randomizing  all  of  the  variables. 
The  study  could  be  supplemented  by  low  cost  assessment  of 
variously  configured  Close  Combat  Tactical  Trainer  prototypes 
for  device  to  device  comparisons. 

The  GAO  states  the  facts  correctly.  However,  the  GAO’s 
conclusion  that  a  limitation  exists  because  the  test  plan  does 
not  include  a  Close  combat  Tactical  Trainer  exercise  compared  to 
a  similar  field  exercise  scenario  is  flawed.  The  test  plan  does 
not  include  this  comparison  because  it  is  not  part  of  our 
training  strategy.  The  Close  Combat  Tactical  Trainer  is 
designed  to  be  used  as  a  pre-field  and  a  post-field  trainer  It 
is  not  being  designed  as  a  replacement  for  actual  field 
training.  Therefore,  there  is  no  justification  to  execute  a 
comparative  test. 


MHPIM8  0; 


The  GAO  reported  that,  because  of  pressures  within  the  Army  to 
have  networked  simulator  capabilities  that  were  greater  than 
those  available  through  Simulation  Networking,  the  Army  decided 
to  field  in  the  U.S.  and  Germany  68  limited-capability  "Quick 
Start*  Close  Combat  Tactical  Trainer  units  in  advance  of  formal 
system  testing  and  a  formal  production  decision.  The  GAO 
observed  that,  as  a  result,  those  units  will  be  fielded  more 
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properly  staffed  to  carry  out  its  assigned  responsibilities. (p. 
Now  on  pp.  5  and  25.  8,  p.  40/gao  Draft  Report) 

pod  response:  Concur.  Action  has  been  initiated  to  obtain 
permanent  personnel  billets.  A  military  (06)  billet  has  been 
provided  and  will  be  used  for  the  DMSO  Director. 


•  nxeQHMEMDATlOH  2 1  The  GAO  recommended  that  the  Secretary  of 
Defense  eliminate  the  duplication  and  inefficiencies  within  the 
two  Joint  Staff  directorates  involved  in  simulations  and  clearly 
delineate  each  directorate's  roles  and  responsibilities  in  this 
Now  on  pp.  5  and  25.  area.  (p.  8,  p.  40/GAO  Draft  Report) 

pod  RRSPONBH :  partially  concur.  Joint  Administrative 
Publication  1.1,  ‘Organization  of  and  Functions  of  the  Joint 
Staff  ,  •  currently  assigns  modeling  and  simulation 
responsibility/proponency  to  several  Joint  Staff  directorates, 
including  J-4  (Logistics),  J-6  (Command  and  Control),  J-7 
(Interoperability  and  Training) ,  and  J-8  (Force  Structure  and 
Assessment) .  Directorate  modeling  and  simulation  activity 
within  those  assigned  areas  is  appropriate  and  does  not 
constitute  unjustified  duplication  or  overlap  of  effort. 

The  Joint  Staff  manages  modeling  and  simulation  activities  as  an 
integral  part  of  its  Information  Resources  Management  Program. 
The  Vice  Director,  Joint  Staff  is  the  Joint  Staff  senior 
Information  Resources  Management  authority  and  has  delegated  to 
the  Deputy  Director  for  Technical  Operations.  J-8,  the 
responsibility  for  coordination,  planning,  and  management  of 
modeling  and  simulation  technology  and  technical  operations. 

The  GAO  report  accurately  reflects  the  status  of  ongoing  Joint 
Staff  studies  that  will  clarify  the  use  and  application  of 
modeling  and  simulation  in  support  of  the  Chairman  of  the  Joint 
Chiefs  of  Staff,  the  Joint  Staff,  and  the  Unified  and  Specified 
Commanders;  so  as  to  maximize  the  utilization  of  ever  scarcer 
modeling  and  simulation  resources.  The  first  study  on  the  roles 
and  missions  of  the  Chairman  was  completed  in  February  1993 . 

The  second  study  on  the  establishment  of  a  Joint  Service 
training,  doctrine,  and  simulation  center  is  ongoing  and  will  be 
completed  in  June  1993.  The  final  study  on  ths  reorganizstion 
of  the  Joint  Staff  was  started  in  March  1993  and  is  expected  to 
be  completed  by  June  1993. 
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be  completed  by  June  1993. 


•  RgCOlOtBNPATlOM  3 ;  The  GAO  recommended  that  the  Secretary  of 
the  Army  ensure  that  all  prerequisite  testing,  cost  analyses, 
and  training  effectiveness  assessments  are  completed  and  fully 
considered  before  decisions  are  made  about  full  production  of 
the  Close  Combat  Tactical  Trainer,  (p.  8,  p.  64/GAO  Draft 
Report ) 

POD  RESPONSE :  Concur.  Of  the  two  types  of  required  testing 
(technical  and  operational)  ,  technical  testing  is  straight 
forward  and  can  be  executed  using  established  techniques.  The 
intent  of  technical  testing  is  to  ensure  the  system  complies 
with  the  requirements  established  by  the  user  and  implemented  by 
the  developer  in  the  specification.  The  Close  Combat  Tactical 
Trainer  will  have  all  requisite  technical  testing  performed. 
Technical  testing  is  scheduled  to  start  July  1996. 

The  operational  testing,  however,  includes  an  unprecedented, 
critical  issue  on  training  transfer.  Methodology  must  be 
developed  to  determine  training  transfer  in  a  combined  arms 
environment  because  of  the  myriad  of  variables  involved  in 
combined  arms  operations.  This  baseline  will  provide  data  to 
determine  the  magnitude  of  the  effort  required  to  establish  the 
methodology  to  be  employed  on  Close  Combat  Tactical  Trainer. 
Operational  testing  is  scheduled  to  begin  December  1996. 

All  technical  and  operational  tests  will  be  completed  by  April 
1997,  prior  to  Milestone  III,  full  Close  Combat  Tactical  Trainer 
production. 
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The  following  are  gao’s  comments  on  the  Department  of  Defense’s  letter 
dated  March  26, 1993. 


GAO  Comments 


1.  We  have  modified  the  report  as  appropriate. 

2.  We  recognize,  as  stated  in  our  report,  that  simulations  have  wide 
applicability  within  dod  beyond  training,  including  the  areas  of  education, 
military  operations,  research  and  development,  test  and  evaluation, 
analysis,  and  production  and  logistics.  However,  the  focus  of  this  report 
was  limited  to  training. 

3.  dod’s  comments  should  not  be  interpreted  as  meaning  that  the 
Combined  Arms  Training  Strategy  is  complete  and  capable  of  identifying 
the  appropriate  mix  of  field  and  simulation  training.  As  noted  in  chapter  4, 
the  strategy  has  not  yet  been  finalized.  Further,  it  is  not  clear  when 
definitive  data  will  be  available  to  provide  precise  answers  to  questions 
about  the  most  appropriate  mix  of  field  and  simulation  training.  Such 
answers  would  likely  vary  for  individual  simulation  systems,  dod 
concurred,  without  additional  comment  regarding  our  assessment  of  the 
training  strategy  contained  in  chapter  4. 

4.  dod’s  position  that  the  cctt  is  designed  to  be  used  as  a  pre-field  and  a 
post-field  trainer  suggests  that  a  ccrr-only  test  scenario  might  provide  a 
clearer  indication  of  the  system’s  capabilities  as  a  pre-field  trainer. 
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